2026-04-02 10:21:40,849 [ 309205 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-04-02 10:21:40,850 [ 309205 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-04-02 10:21:40,850 [ 309205 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-04-02 10:21:40,850 [ 309205 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_6biipx --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke test_row_policy/test.py::test_some_users_without_policies test_row_policy/test.py::test_tags_with_db_and_table_names test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data test_row_policy/test.py::test_users_xml_is_readonly test_row_policy/test.py::test_with_prewhere test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_storage_hudi/test.py::test_multiple_hudi_files test_storage_hudi/test.py::test_single_hudi_file test_storage_hudi/test.py::test_types test_storage_redis/test.py::test_create_table test_storage_redis/test.py::test_delete test_storage_redis/test.py::test_select_int test_storage_redis/test.py::test_simple_insert test_storage_redis/test.py::test_simple_select test_storage_redis/test.py::test_truncate test_storage_redis/test.py::test_update test_storage_s3_queue/test.py::test_commit_on_limit 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered]' 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: order-1.0.1, random-0.2, timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_session_log/test.py::test_grpc_session test_reloading_storage_configuration/test.py::test_add_disk test_storage_redis/test.py::test_create_table test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_rocksdb_options/test.py::test_invalid_block_based_table_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] test_storage_hudi/test.py::test_multiple_hudi_files test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_recovery_replica/test.py::test_choose_source_replica [gw5] [ 1%] PASSED test_storage_redis/test.py::test_create_table test_storage_redis/test.py::test_delete [gw8] [ 2%] PASSED test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session [gw4] [ 3%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] [gw5] [ 4%] PASSED test_storage_redis/test.py::test_delete test_storage_redis/test.py::test_select_int [gw5] [ 5%] PASSED test_storage_redis/test.py::test_select_int test_storage_redis/test.py::test_simple_insert [gw0] [ 6%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw8] [ 7%] PASSED test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions [gw9] [ 8%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy [gw5] [ 9%] PASSED test_storage_redis/test.py::test_simple_insert test_storage_redis/test.py::test_simple_select [gw1] [ 10%] PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with [gw5] [ 11%] PASSED test_storage_redis/test.py::test_simple_select test_storage_redis/test.py::test_truncate [gw2] [ 12%] PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options [gw4] [ 13%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] [gw9] [ 14%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list [gw5] [ 15%] PASSED test_storage_redis/test.py::test_truncate test_storage_redis/test.py::test_update test_row_policy/test.py::test_change_of_users_xml_changes_row_policies [gw0] [ 16%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw5] [ 17%] PASSED test_storage_redis/test.py::test_update [gw9] [ 18%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy [gw8] [ 19%] PASSED test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch [gw6] [ 20%] PASSED test_recovery_replica/test.py::test_choose_source_replica [gw9] [ 21%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_recovery_replica/test.py::test_recovery [gw4] [ 22%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] [gw8] [ 23%] PASSED test_session_log/test.py::test_postgres_session [gw0] [ 24%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy [gw9] [ 25%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy [gw4] [ 26%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_storage_s3_queue/test.py::test_commit_on_limit [gw9] [ 27%] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy [gw4] [ 28%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw6] [ 29%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw0] [ 30%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy [gw1] [ 31%] PASSED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies [gw5] [ 32%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw4] [ 33%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw5] [ 34%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_row_policy/test.py::test_dcl_introspection [gw0] [ 35%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_settings_constraints_distributed/test.py::test_insert_clamps_settings [gw9] [ 36%] PASSED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw0] [ 37%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_read_temporary_tables_on_failure/test.py::test_different_versions [gw1] [ 38%] PASSED test_row_policy/test.py::test_dcl_introspection [gw4] [ 39%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_row_policy/test.py::test_dcl_management test_reload_query_masking_rules/test.py::test_reload_query_masking_rules [gw0] [ 40%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy [gw9] [ 41%] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions [gw1] [ 42%] PASSED test_row_policy/test.py::test_dcl_management [gw2] [ 43%] FAILED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_replace_partition/test.py::test_normal_work test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw1] [ 44%] PASSED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw5] [ 45%] PASSED test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings [gw3] [ 46%] FAILED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] [gw3] [ 47%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format [gw4] [ 48%] PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_row_policy/test.py::test_grant_create_row_policy [gw0] [ 49%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings [gw1] [ 50%] PASSED test_row_policy/test.py::test_grant_create_row_policy [gw2] [ 51%] FAILED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options [gw4] [ 52%] PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings [gw9] [ 53%] PASSED test_replace_partition/test.py::test_normal_work test_row_policy/test.py::test_introspection [gw1] [ 54%] PASSED test_row_policy/test.py::test_introspection [gw6] [ 55%] PASSED test_recovery_replica/test.py::test_update_metadata test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage [gw5] [ 56%] PASSED test_settings_constraints_distributed/test.py::test_select_clamps_settings [gw8] [ 57%] PASSED test_storage_s3_queue/test.py::test_commit_on_limit test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] test_row_policy/test.py::test_join [gw1] [ 58%] PASSED test_row_policy/test.py::test_join test_replication_without_zookeeper/test.py::test_startup_without_zookeeper [gw3] [ 59%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] [gw3] [ 60%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] [gw8] [ 61%] PASSED test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] test_part_uuid/test.py::test_part_uuid test_row_policy/test.py::test_miscellaneous_engines test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl [gw8] [ 62%] PASSED test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] [gw4] [ 63%] SKIPPED test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage [gw1] [ 64%] PASSED test_row_policy/test.py::test_miscellaneous_engines [gw3] [ 65%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] [gw2] [ 66%] FAILED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw7] [ 67%] FAILED test_storage_hudi/test.py::test_multiple_hudi_files test_storage_hudi/test.py::test_single_hudi_file test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned [gw9] [ 68%] FAILED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper [gw1] [ 69%] PASSED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side [gw1] [ 70%] PASSED test_row_policy/test.py::test_policy_on_distributed_table_via_role [gw6] [ 71%] PASSED test_part_uuid/test.py::test_part_uuid [gw3] [ 72%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] test_row_policy/test.py::test_reload_users_xml_by_timer [gw5] [ 73%] PASSED test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl test_reload_zookeeper/test.py::test_reload_zookeeper [gw3] [ 74%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] [gw9] [ 75%] PASSED test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side [gw1] [ 76%] PASSED test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery [gw1] [ 77%] PASSED test_row_policy/test.py::test_row_policy_filter_with_subquery [gw3] [ 78%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format [gw0] [ 79%] FAILED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_row_policy/test.py::test_smoke [gw2] [ 80%] FAILED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options [gw1] [ 81%] PASSED test_row_policy/test.py::test_smoke [gw3] [ 82%] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_row_policy/test.py::test_some_users_without_policies [gw6] [ 83%] PASSED test_reload_zookeeper/test.py::test_reload_zookeeper [gw1] [ 84%] PASSED test_row_policy/test.py::test_some_users_without_policies test_row_policy/test.py::test_tags_with_db_and_table_names [gw1] [ 85%] PASSED test_row_policy/test.py::test_tags_with_db_and_table_names test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter [gw0] [ 86%] FAILED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk [gw1] [ 87%] PASSED test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter [gw2] [ 88%] FAILED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw2] [ 89%] FAILED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw2] [ 90%] FAILED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw2] [ 91%] FAILED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw2] [ 92%] FAILED test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter [gw1] [ 93%] PASSED test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter [gw7] [ 94%] FAILED test_storage_hudi/test.py::test_single_hudi_file test_storage_hudi/test.py::test_types [gw7] [ 95%] FAILED test_storage_hudi/test.py::test_types test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data [gw1] [ 96%] PASSED test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data [gw1] [ 97%] PASSED test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data test_row_policy/test.py::test_users_xml_is_readonly [gw1] [ 98%] PASSED test_row_policy/test.py::test_users_xml_is_readonly test_row_policy/test.py::test_with_prewhere [gw1] [ 99%] PASSED test_row_policy/test.py::test_with_prewhere [gw0] [100%] FAILED test_reloading_storage_configuration/test.py::test_update_disk =================================== FAILURES =================================== ______________________ test_invalid_column_family_options ______________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_invalid_column_family_options(start_cluster): node.exec_in_container( [ "bash", "-c", "sed -i 's/num_levels/no_such_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml", ] ) > node.restart_clickhouse() test_rocksdb_options/test.py:135: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:24:38 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/num_levels/no_such_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:38 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "sed -i 's/num_levels/no_such_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:113, run_and_check) 2026-04-02 10:24:40 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:40 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:24:42 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:24:42 [ 334 ] DEBUG : Stdout: 1565 ? 00:00:31 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:24:42 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:42 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:24:47 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:47 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:24:50 [ 334 ] DEBUG : Stdout:1565 (cluster.py:121, run_and_check) 2026-04-02 10:24:51 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:51 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:24:54 [ 334 ] DEBUG : Stdout:1565 (cluster.py:121, run_and_check) 2026-04-02 10:24:55 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:55 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:24:58 [ 334 ] DEBUG : Stdout:1565 (cluster.py:121, run_and_check) 2026-04-02 10:24:59 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:59 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:01 [ 334 ] DEBUG : Stdout:1565 (cluster.py:121, run_and_check) 2026-04-02 10:25:02 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:02 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:03 [ 334 ] DEBUG : Stdout:1565 (cluster.py:121, run_and_check) 2026-04-02 10:25:03 [ 334 ] DEBUG : Stdout:2348 (cluster.py:121, run_and_check) 2026-04-02 10:25:04 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:04 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:05 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:05 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:06 [ 334 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3964, start_clickhouse) 2026-04-02 10:25:06 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:06 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2026-04-02 10:25:12 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:12 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:13 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:25:13 [ 334 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:25:13 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:13 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:15 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:25:15 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:17 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:20 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:23 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:25 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:27 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:32 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:35 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:36 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:39 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:41 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:41 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:44 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:25:44 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:25:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:25:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:25:51 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:25:51 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:53 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:56 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:25:58 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:00 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:01 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:04 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:07 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:09 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:11 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:26:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:26:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:26:18 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:26:18 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:26:18 [ 334 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:26:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:26:18 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:26:20 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:26:20 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:30 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:26:20 [ 334 ] INFO : PS RESULT: PID TTY TIME CMD 2385 ? 00:00:30 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:26:20 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:26:20 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:26:27 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:26:27 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:2173, exec_in_container) 2026-04-02 10:26:27 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:113, run_and_check) 2026-04-02 10:31:28 [ 334 ] WARNING : Current start attempt failed. Will kill 2385 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:31:28 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2385'] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:28 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'kill -9 2385'] (cluster.py:113, run_and_check) ______________________________ test_log_table[s3] ______________________________ [gw3] linux -- Python 3.10.12 /usr/bin/python3 cluster = storage_policy = 's3' @pytest.mark.parametrize("storage_policy", ["s3", "s3_plain"]) def test_log_table(cluster, storage_policy): if storage_policy == "s3_plain": # Log table doesn't work on s3_plain. Rename operation is not implemented return node = cluster.instances["switching_node"] create_table_statement = f""" CREATE TABLE test_log_table ( id Int64, val String ) ENGINE=Log SETTINGS storage_policy='{storage_policy}' """ node.query(create_table_statement) node.query("INSERT INTO test_log_table VALUES (0, 'a')") assert "1" == node.query("SELECT count() FROM test_log_table").strip() > switch_config_write_full_object_key(node, True) test_remote_blobs_naming/test_backward_compatibility.py:448: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_remote_blobs_naming/test_backward_compatibility.py:423: in switch_config_write_full_object_key node.restart_clickhouse() helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-04-02 10:21:56 [ 337 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 337 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-02 10:21:56 [ 337 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-04-02 10:21:56 [ 337 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-04-02 10:21:56 [ 337 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 337 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 337 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV HOSTNAME 5c0c783dd6f2 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV SHLVL 0 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV HOME /root (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV OLDPWD / (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_HELPER_TAG 2cffe1eae894 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke test_row_policy/test.py::test_some_users_without_policies test_row_policy/test.py::test_tags_with_db_and_table_names test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data test_row_policy/test.py::test_users_xml_is_readonly test_row_policy/test.py::test_with_prewhere test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_storage_hudi/test.py::test_multiple_hudi_files test_storage_hudi/test.py::test_single_hudi_file test_storage_hudi/test.py::test_types test_storage_redis/test.py::test_create_table test_storage_redis/test.py::test_delete test_storage_redis/test.py::test_select_int test_storage_redis/test.py::test_simple_insert test_storage_redis/test.py::test_simple_select test_storage_redis/test.py::test_truncate test_storage_redis/test.py::test_update test_storage_s3_queue/test.py::test_commit_on_limit 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered]' 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered]' -vvv (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_USE_OLD_ANALYZER 1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV TZ Etc/UTC (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV DOCKER_BASE_TAG 1e0b53d756cf (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 0 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV WORKER_FREE_PORTS 30150 30151 30152 30153 30154 30155 30156 30157 30158 30159 30160 30161 30162 30163 30164 30165 30166 30167 30168 30169 30170 30171 30172 30173 30174 30175 30176 30177 30178 30179 30180 30181 30182 30183 30184 30185 30186 30187 30188 30189 30190 30191 30192 30193 30194 30195 30196 30197 30198 30199 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTEST_XDIST_TESTRUNUID 835aeb6757d841b8831c932b29425f1c (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTEST_XDIST_WORKER gw3 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : ENV PYTEST_CURRENT_TEST test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] (setup) (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:766, __init__) 2026-04-02 10:21:56 [ 337 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1785, add_instance) 2026-04-02 10:21:56 [ 337 ] DEBUG : Setup Keeper (cluster.py:1057, setup_keeper_cmd) 2026-04-02 10:21:56 [ 337 ] DEBUG : Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility_gw3. Added instance name:node tag:1e0b53d756cf base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2079, add_instance) 2026-04-02 10:21:56 [ 337 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1785, add_instance) 2026-04-02 10:21:56 [ 337 ] DEBUG : Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility_gw3. Added instance name:new_node tag:1e0b53d756cf base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2079, add_instance) 2026-04-02 10:21:56 [ 337 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1785, add_instance) 2026-04-02 10:21:56 [ 337 ] DEBUG : Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility_gw3. Added instance name:switching_node tag:1e0b53d756cf base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2079, add_instance) 2026-04-02 10:21:56 [ 337 ] INFO : Starting cluster... (test_backward_compatibility.py:55, cluster) 2026-04-02 10:21:56 [ 337 ] INFO : Running tests in /ClickHouse/tests/integration/test_remote_blobs_naming/test_backward_compatibility.py (cluster.py:2788, start) 2026-04-02 10:21:56 [ 337 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start) 2026-04-02 10:21:56 [ 337 ] DEBUG : Docker networks for project roottestremoteblobsnamingbackwardcompatibility_gw3 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Docker containers for project roottestremoteblobsnamingbackwardcompatibility_gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Docker volumes for project roottestremoteblobsnamingbackwardcompatibility_gw3 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:21:57 [ 337 ] DEBUG : Docker networks for project roottestremoteblobsnamingbackwardcompatibility_gw3 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Docker containers for project roottestremoteblobsnamingbackwardcompatibility_gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Docker volumes for project roottestremoteblobsnamingbackwardcompatibility_gw3 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:21:57 [ 337 ] DEBUG : Command:docker container list --all --filter name='^/roottestremoteblobsnamingbackwardcompatibility_gw3_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:21:57 [ 337 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:21:57 [ 337 ] DEBUG : No running containers for project: roottestremoteblobsnamingbackwardcompatibility_gw3 (cluster.py:904, cleanup) 2026-04-02 10:21:57 [ 337 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:21:58 [ 337 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:21:58 [ 337 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:21:58 [ 337 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:21:58 [ 337 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:21:58 [ 337 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:21:58 [ 337 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:21:58 [ 337 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup directory for instance: node (cluster.py:2808, start) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/old_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/database (cluster.py:4649, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/logs (cluster.py:4660, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup directory for instance: new_node (cluster.py:2808, start) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/new_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf_new.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/database (cluster.py:4649, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/logs (cluster.py:4660, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup directory for instance: switching_node (cluster.py:2808, start) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/switching_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/database (cluster.py:4649, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/logs (cluster.py:4660, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4746, create_dir) 2026-04-02 10:21:58 [ 337 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:1e0b53d756cf', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/certs/public.crt'} stored in /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env (cluster.py:86, _create_env_file) 2026-04-02 10:21:58 [ 337 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:21:58 [ 337 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:21:58 [ 337 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:21:58 [ 337 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:21:58 [ 337 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request) 2026-04-02 10:21:58 [ 337 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy2 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo3 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling minio1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling resolver ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo2 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling switching_node ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling new_node ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling node ... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy2 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling switching_node ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy1 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling new_node ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy2 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy1 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo2 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy2 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling switching_node ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling switching_node ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling proxy1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo2 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling new_node ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling new_node ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling switching_node ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling new_node ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling node ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling minio1 ... pulling from minio/minio (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling resolver ... pulling from altinityinfra/python... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling node ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling node ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling minio1 ... digest: sha256:9eea83a4e1425067e7... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling minio1 ... status: image is up to date for m... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling node ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo3 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling minio1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling resolver ... digest: sha256:919cb0e6c1a4f05e8d... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling resolver ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo3 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling resolver ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Stderr:Pulling zoo1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:23 [ 337 ] DEBUG : Setup ZooKeeper (cluster.py:2849, start) 2026-04-02 10:22:23 [ 337 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/coordination', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/coordination'] (cluster.py:2850, start) 2026-04-02 10:22:23 [ 337 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--verbose', 'up', '-d'] (cluster.py:113, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:docker-py version: (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-130-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.25', 'Details': {'GitCommit': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}}, {'Name': 'runc', 'Version': '1.2.4', 'Details': {'GitCommit': 'v1.2.4-0-g6c52b3f'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-130-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestremoteblobsnamingbackwardcompatibilitygw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'ID': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Containers': 12, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.network.ensure: Creating network "roottestremoteblobsnamingbackwardcompatibility_gw3_default" with the default driver (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='roottestremoteblobsnamingbackwardcompatibility_gw3_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': '78fe07f357e24a4fea19ec89a97f1b909f895f9adf617232c2360ec1954835c6', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , } (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo2', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo3', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo3', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 1333b3b42c67f844d5e8cfc828d00a8360a2327ad68cc3c78f809cba26a2fc0e (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/log', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Target': '/var/log/clickhouse-keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config1.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_zoo1_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'zoo1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '1333b3b42c67f844d5e8cfc828d00a8360a2327ad68cc3c78f809cba26a2fc0e'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['zoo1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 8e3c8a67e86c412f5315fd75a51f4737df5294f8b62a6cdbfb9e75be0e7ee1bc (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Target': '/var/lib/clickhouse', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config3.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'zoo3', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '8e3c8a67e86c412f5315fd75a51f4737df5294f8b62a6cdbfb9e75be0e7ee1bc'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['zoo3'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 9a25039526ccfa2c68ab43a8c81c7ca94d7de02b7f25c9eaf77a920a4111f96f (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Target': '/var/lib/clickhouse', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config2.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_zoo2_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'zoo2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '9a25039526ccfa2c68ab43a8c81c7ca94d7de02b7f25c9eaf77a920a4111f96f'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['zoo2'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config1.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config3.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['926e448ac431', 'zoo1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['zoo3', '144a65d43317'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config2.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['858beac185e4', 'zoo2'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a') (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='zoo3', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 337 ] DEBUG : Wait ZooKeeper to start (cluster.py:2504, wait_zookeeper_to_start) 2026-04-02 10:22:34 [ 337 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:34 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_zoo1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:34 [ 337 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.9.2, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:22:35 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:35 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:35 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:35 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:35 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:35 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:35 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:35 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:36 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:36 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:37 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:37 [ 337 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] INFO : Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:22:39 [ 337 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:39 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_zoo2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:39 [ 337 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.9.4, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:22:39 [ 337 ] INFO : Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:22:39 [ 337 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:39 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:39 [ 337 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.9.3, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:22:39 [ 337 ] INFO : Connecting to 172.16.9.3(172.16.9.3):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:22:39 [ 337 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:22:39 [ 337 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:22:39 [ 337 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:22:39 [ 337 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:22:39 [ 337 ] DEBUG : All instances of ZooKeeper Secure started (cluster.py:2519, wait_zookeeper_nodes_to_start) 2026-04-02 10:22:39 [ 337 ] INFO : Trying to create Minio instance by command docker-compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env --project-name roottestremoteblobsnamingbackwardcompatibility_gw3 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:3087, start) 2026-04-02 10:22:39 [ 337 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--verbose', 'up', '-d'] (cluster.py:113, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:docker-py version: (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-130-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.25', 'Details': {'GitCommit': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}}, {'Name': 'runc', 'Version': '1.2.4', 'Details': {'GitCommit': 'v1.2.4-0-g6c52b3f'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-130-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestremoteblobsnamingbackwardcompatibilitygw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('roottestremoteblobsnamingbackwardcompatibilitygw3_data1-1') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'ID': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Containers': 32, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'ConfigFrom': {'Network': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'ConfigOnly': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Containers': {'144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3': {'EndpointID': '5885d987b6919d68835c8538af5cc74c19481b401bacb47b59f7b0055e43bff3', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'IPv4Address': '172.16.9.3/24', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'IPv6Address': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'MacAddress': '02:42:ac:10:09:03', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Name': 'roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a': {'EndpointID': '6fe5759a80b7277aa01654753367fc91c29450308d7e59b31f271032aba6f9cb', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'IPv4Address': '172.16.9.4/24', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.volume.initialize: Creating volume "roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1" with default driver (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume <- ('roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1', None, None, labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.volume': 'data1-1', 'com.docker.compose.version': '1.29.2'}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume -> {'CreatedAt': '2026-04-02T10:22:42Z', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Driver': 'local', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Labels': {'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'com.docker.compose.version': '1.29.2', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'com.docker.compose.volume': 'data1-1'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Mountpoint': '/var/lib/docker/volumes/roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1/_data', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Name': 'roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Options': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Scope': 'local'} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 3 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 3 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('144a65d43317498ae2d128ebaf287ffc38b3311806ae8784b5be5713c3e3d3d3') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config3.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('858beac185e4dec49e2385dcaac17ea97145131ec02b83a320e92cf964b3502a') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config2.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('926e448ac4314e2bb3e516996d1fbd21e95db4f6719eb6d758e9d8b46a43ece0') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config1.xml', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.project.find_orphan_containers: Found orphan containers (roottestremoteblobsnamingbackwardcompatibility_gw3_zoo3_1, roottestremoteblobsnamingbackwardcompatibility_gw3_zoo2_1, roottestremoteblobsnamingbackwardcompatibility_gw3_zoo1_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.project._get_convergence_plans: minio1 has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.project._get_convergence_plans: resolver has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , , } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_proxy2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy2', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_proxy1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 6b5606ca225ecc515aa0ef997df4536f0cfdb9d53543b023f13743c53b4861f4 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_proxy2_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'proxy2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '6b5606ca225ecc515aa0ef997df4536f0cfdb9d53543b023f13743c53b4861f4'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['proxy2'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 6b5606ca225ecc515aa0ef997df4536f0cfdb9d53543b023f13743c53b4861f4 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_proxy1_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'proxy1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '6b5606ca225ecc515aa0ef997df4536f0cfdb9d53543b023f13743c53b4861f4'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['proxy1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '5b1bc6ae21513cb0d70306565b2a3ebbda5a03b5686e585e0e34bd035b8c13d7', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('5b1bc6ae21513cb0d70306565b2a3ebbda5a03b5686e585e0e34bd035b8c13d7') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '529e1426bd5e9fb231379ba05e28d9d05f776ae79834d7f168455c419601a806', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('529e1426bd5e9fb231379ba05e28d9d05f776ae79834d7f168455c419601a806') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('5b1bc6ae21513cb0d70306565b2a3ebbda5a03b5686e585e0e34bd035b8c13d7', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('529e1426bd5e9fb231379ba05e28d9d05f776ae79834d7f168455c419601a806', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('529e1426bd5e9fb231379ba05e28d9d05f776ae79834d7f168455c419601a806', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['proxy1', '529e1426bd5e'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('529e1426bd5e9fb231379ba05e28d9d05f776ae79834d7f168455c419601a806') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('5b1bc6ae21513cb0d70306565b2a3ebbda5a03b5686e585e0e34bd035b8c13d7', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['proxy2', '5b1bc6ae2151'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('5b1bc6ae21513cb0d70306565b2a3ebbda5a03b5686e585e0e34bd035b8c13d7') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestremoteblobsnamingbackwardcompatibilitygw3', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_resolver_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='resolver', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_minio1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='minio1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: dd4f6320b87b4179a69a69650fb099a6b7cf7b9582c9ebc24d364dfd927f6ee8 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=['roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/certs:/certs:rw'], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1:/data1:rw', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/certs:/certs:rw'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='server --console-address 127.0.0.1:19001 --address :9001 --certs-dir /certs /data1-1', environment=['MINIO_ACCESS_KEY=minio', 'MINIO_SECRET_KEY=minio123', 'MINIO_PROMETHEUS_AUTH_TYPE=public'], image='minio/minio:RELEASE.2024-07-31T05-46-26Z', volumes={'/data1': {}, '/certs': {}}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_minio1_1', detach=True, ports=['9001'], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'minio1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'dd4f6320b87b4179a69a69650fb099a6b7cf7b9582c9ebc24d364dfd927f6ee8'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'VolumesFrom': [], 'Binds': ['roottestremoteblobsnamingbackwardcompatibility_gw3_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/minio/certs:/certs:rw'], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['minio1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: f415db201dfea648be3340e6de31535e805a8df1c766d40444222a41974a9f14 (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestremoteblobsnamingbackwardcompatibility_gw3_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/python-bottle:caad4729259e', tty=True, volumes={}, name='roottestremoteblobsnamingbackwardcompatibility_gw3_resolver_1', detach=True, ports=['8080'], environment=[], labels={'com.docker.compose.project': 'roottestremoteblobsnamingbackwardcompatibility_gw3', 'com.docker.compose.service': 'resolver', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'f415db201dfea648be3340e6de31535e805a8df1c766d40444222a41974a9f14'}, host_config={'NetworkMode': 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'roottestremoteblobsnamingbackwardcompatibility_gw3_default': {'Aliases': ['resolver'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '160387e69621834803534d7db2c3f46339f2b916c14cc0093cf1e29edbbd4509', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('160387e69621834803534d7db2c3f46339f2b916c14cc0093cf1e29edbbd4509') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '5242cf7f26dda74fda2611917545148bc58682c292454dcd1b6325ca3b31bd63', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('5242cf7f26dda74fda2611917545148bc58682c292454dcd1b6325ca3b31bd63') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('160387e69621834803534d7db2c3f46339f2b916c14cc0093cf1e29edbbd4509', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Args': ['server', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--console-address', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '127.0.0.1:19001', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--address', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: ':9001', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '--certs-dir', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '/certs', (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: '/data1-1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('5242cf7f26dda74fda2611917545148bc58682c292454dcd1b6325ca3b31bd63', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('5242cf7f26dda74fda2611917545148bc58682c292454dcd1b6325ca3b31bd63', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['5242cf7f26dd', 'minio1'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('160387e69621834803534d7db2c3f46339f2b916c14cc0093cf1e29edbbd4509', 'roottestremoteblobsnamingbackwardcompatibility_gw3_default', aliases=['resolver', '160387e69621'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('5242cf7f26dda74fda2611917545148bc58682c292454dcd1b6325ca3b31bd63') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('160387e69621834803534d7db2c3f46339f2b916c14cc0093cf1e29edbbd4509') (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_resolver_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestremoteblobsnamingbackwardcompatibility_gw3', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_minio1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:59 [ 337 ] INFO : Trying to connect to Minio... (cluster.py:3093, start) 2026-04-02 10:22:59 [ 337 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:59 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_minio1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:59 [ 337 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:59 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_proxy1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:59 [ 337 ] DEBUG : Starting new HTTP connection (1): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:59 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:22:59 [ 337 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:22:59 [ 337 ] DEBUG : Starting new HTTP connection (2): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:00 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:23:00 [ 337 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:23:00 [ 337 ] DEBUG : Starting new HTTP connection (3): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:00 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:23:00 [ 337 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:23:00 [ 337 ] DEBUG : Starting new HTTP connection (4): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:00 [ 337 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.9.7', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2669, wait_minio_to_start) 2026-04-02 10:23:01 [ 337 ] DEBUG : Starting new HTTP connection (5): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:01 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:23:01 [ 337 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:23:01 [ 337 ] DEBUG : Starting new HTTP connection (6): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:01 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:23:01 [ 337 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:23:01 [ 337 ] DEBUG : Starting new HTTP connection (7): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:01 [ 337 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:23:01 [ 337 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:23:01 [ 337 ] DEBUG : Starting new HTTP connection (8): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:01 [ 337 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.9.7', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2669, wait_minio_to_start) 2026-04-02 10:23:02 [ 337 ] DEBUG : Starting new HTTP connection (9): 172.16.9.7:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:23:02 [ 337 ] DEBUG : http://172.16.9.7:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:02 [ 337 ] DEBUG : Connected to Minio. (cluster.py:2649, wait_minio_to_start) 2026-04-02 10:23:02 [ 337 ] DEBUG : http://172.16.9.7:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:02 [ 337 ] DEBUG : http://172.16.9.7:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:02 [ 337 ] DEBUG : S3 bucket 'root' created (cluster.py:2664, wait_minio_to_start) 2026-04-02 10:23:02 [ 337 ] DEBUG : http://172.16.9.7:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:02 [ 337 ] DEBUG : http://172.16.9.7:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:02 [ 337 ] DEBUG : S3 bucket 'root2' created (cluster.py:2664, wait_minio_to_start) 2026-04-02 10:23:02 [ 337 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env --project-name roottestremoteblobsnamingbackwardcompatibility_gw3 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/docker-compose.yml up -d --no-recreate') (cluster.py:3146, start) 2026-04-02 10:23:02 [ 337 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility_gw3', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances_backward_compatibility_0_gw3/switching_node/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_new_node_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_node_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_node_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_new_node_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : Stderr:Creating roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:23:16 [ 337 ] DEBUG : ClickHouse instance created (cluster.py:3154, start) 2026-04-02 10:23:16 [ 337 ] DEBUG : get_instance_ip instance_name=node (cluster.py:2135, get_instance_ip) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : Waiting for ClickHouse start in node, ip: 172.16.9.10... (cluster.py:3161, start) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:23 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:24 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:25 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:26 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:27 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:28 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:28 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:28 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:28 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:28 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:29 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e2334a36944842a809d797c1092ec850e36025eb18fc9e0866a5e7db549a09a9/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : ClickHouse node started (cluster.py:3165, start) 2026-04-02 10:23:30 [ 337 ] DEBUG : get_instance_ip instance_name=new_node (cluster.py:2135, get_instance_ip) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_new_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : Waiting for ClickHouse start in new_node, ip: 172.16.9.11... (cluster.py:3161, start) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_new_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:30 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/e8bde6cd81fa363ee9ffbbbc50f07b72f79f9c0bf554d238d9bd68db92621072/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : ClickHouse new_node started (cluster.py:3165, start) 2026-04-02 10:23:31 [ 337 ] DEBUG : get_instance_ip instance_name=switching_node (cluster.py:2135, get_instance_ip) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : Waiting for ClickHouse start in switching_node, ip: 172.16.9.9... (cluster.py:3161, start) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:31 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : http://localhost:None "GET /v1.42/containers/74177fec3b0d7ed4207cf867fe242a28de7bf73df17f365ec3d49728b25a4978/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:32 [ 337 ] DEBUG : ClickHouse switching_node started (cluster.py:3165, start) 2026-04-02 10:23:32 [ 337 ] INFO : Cluster started (test_backward_compatibility.py:57, cluster) ------------------------------ Captured log call ------------------------------- 2026-04-02 10:23:33 [ 337 ] DEBUG : Executing query CREATE TABLE test_log_table ( id Int64, val String ) ENGINE=Log SETTINGS storage_policy='s3' on switching_node (cluster.py:3602, query) 2026-04-02 10:23:39 [ 337 ] DEBUG : Executing query INSERT INTO test_log_table VALUES (0, 'a') on switching_node (cluster.py:3602, query) 2026-04-02 10:23:52 [ 337 ] DEBUG : Executing query SELECT count() FROM test_log_table on switching_node (cluster.py:3602, query) 2026-04-02 10:24:10 [ 337 ] DEBUG : run container_id:roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'cat /etc/clickhouse-server/config.d/switching_node.xml'] (cluster.py:2173, exec_in_container) 2026-04-02 10:24:10 [ 337 ] DEBUG : Command:['docker', 'exec', 'roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1', 'bash', '-c', 'cat /etc/clickhouse-server/config.d/switching_node.xml'] (cluster.py:113, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : Stdout: 0 (cluster.py:121, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : Stdout: (cluster.py:121, run_and_check) 2026-04-02 10:24:14 [ 337 ] DEBUG : run container_id:roottestremoteblobsnamingbackwardcompatibility_gw3_switching_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/0101 def test_invalid_options(start_cluster): node.exec_in_container( [ "bash", "-c", "sed -i 's/max_background_jobs/no_such_option/g' /etc/clickhouse-server/config.d/rocksdb.xml", ] ) > node.restart_clickhouse() test_rocksdb_options/test.py:67: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:31:32 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/max_background_jobs/no_such_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:32 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "sed -i 's/max_background_jobs/no_such_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:113, run_and_check) 2026-04-02 10:31:33 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:33 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:31:34 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:31:34 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:31:34 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:34 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:31:36 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:36 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:38 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:39 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:39 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:40 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:41 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:41 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:42 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:43 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:43 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:44 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:45 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:45 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:47 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:48 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:48 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:49 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:50 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:50 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:51 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:52 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:52 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:53 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:54 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:54 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:55 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:56 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:56 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:57 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:31:58 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:31:58 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:31:59 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:00 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:00 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:01 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:02 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:02 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:03 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:04 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:04 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:06 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:07 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:07 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:07 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:08 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:08 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:10 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:11 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:11 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:12 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:13 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:13 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:15 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:16 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:16 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:18 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:19 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:19 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:20 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:21 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:21 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:22 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:23 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:23 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:24 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:25 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:25 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:27 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:28 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:28 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:29 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:30 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:30 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:32 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:33 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:33 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:34 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:35 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:35 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:36 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:37 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:37 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:40 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:40 [ 334 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:2385 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:32:40 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:40 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:32:45 [ 334 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:32:45 [ 334 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:32:45 [ 334 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:32:45 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:45 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:47 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:47 [ 334 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:32:47 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:47 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:49 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:32:49 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:32:52 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:32:54 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:32:57 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:00 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:04 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:06 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:10 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:12 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:15 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:18 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:20 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:33:20 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:33:20 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:20 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:24 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:33:24 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:26 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:30 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:33 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:36 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:40 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:44 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:48 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:50 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:52 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:33:54 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:54 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:58 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:33:58 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:33:58 [ 334 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:33:58 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:58 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:34:00 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:34:00 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:34:00 [ 334 ] INFO : PS RESULT: PID TTY TIME CMD 2385 ? 00:00:35 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:34:00 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:00 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:02 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:02 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:02 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:113, run_and_check) 2026-04-02 10:34:04 [ 334 ] DEBUG : Stderr:warning: process 2385 is already traced by process 3128process 2385 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:34:04 [ 334 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:34:04 [ 334 ] WARNING : Current start attempt failed. Will kill 2385 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:34:04 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2385'] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:04 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'kill -9 2385'] (cluster.py:113, run_and_check) _________________ test_table_invalid_block_based_table_options _________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_table_invalid_block_based_table_options(start_cluster): node.exec_in_container( [ "bash", "-c", "sed -i 's/format_version/no_such_table_block_based_table_options/g' /etc/clickhouse-server/config.d/rocksdb.xml", ] ) > node.restart_clickhouse() test_rocksdb_options/test.py:219: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:34:10 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/format_version/no_such_table_block_based_table_options/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:10 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "sed -i 's/format_version/no_such_table_block_based_table_options/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:113, run_and_check) 2026-04-02 10:34:16 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:16 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:34:18 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:34:18 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:34:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:18 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:34:21 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:21 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:22 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:23 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:23 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:26 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:27 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:27 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:28 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:29 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:29 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:30 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:31 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:31 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:34 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:35 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:35 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:37 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:38 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:38 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:40 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:41 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:41 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:43 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:47 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:48 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:48 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:50 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:51 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:51 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:54 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:55 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:55 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:34:58 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:34:59 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:34:59 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:02 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:03 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:03 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:10 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:11 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:13 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:15 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:16 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:16 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:17 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:18 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:19 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:20 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:20 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:22 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:23 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:23 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:25 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:25 [ 334 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:2385 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:35:25 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:25 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:35:26 [ 334 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:35:26 [ 334 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:35:26 [ 334 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:35:26 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:26 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:27 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:27 [ 334 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:35:27 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:27 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:30 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:30 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:33 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:35 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:36 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:38 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:39 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:42 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:44 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:45 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:47 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:48 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:48 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:49 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:49 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:35:49 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:35:49 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:35:50 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:35:50 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:51 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:53 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:55 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:56 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:35:58 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:00 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:02 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:03 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:04 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:06 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:06 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:07 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:07 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:36:07 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:07 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:08 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:08 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:09 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:11 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:13 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:15 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:16 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:18 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:19 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:21 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:23 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:25 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:25 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:26 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:26 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:36:26 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:26 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:27 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:27 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:28 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:29 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:31 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:33 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:34 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:36 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:38 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:40 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:42 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:36:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:46 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:46 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:36:46 [ 334 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:36:46 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:46 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:36:47 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:36:47 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:36:47 [ 334 ] INFO : PS RESULT: PID TTY TIME CMD 2385 ? 00:00:35 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:36:47 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:47 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:48 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:48 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:48 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:113, run_and_check) 2026-04-02 10:36:50 [ 334 ] DEBUG : Stderr:warning: process 2385 is already traced by process 3128process 2385 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:36:50 [ 334 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:36:50 [ 334 ] WARNING : Current start attempt failed. Will kill 2385 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:36:50 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2385'] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:50 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'kill -9 2385'] (cluster.py:113, run_and_check) ___________________________ test_multiple_hudi_files ___________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_multiple_hudi_files(started_cluster): instance = started_cluster.instances["node1"] spark = started_cluster.spark_session minio_client = started_cluster.minio_client bucket = started_cluster.minio_bucket TABLE_NAME = "test_multiple_hudi_files" write_hudi_from_df( spark, TABLE_NAME, generate_data(spark, 0, 100), f"/{TABLE_NAME}" ) files = upload_directory(minio_client, bucket, f"/{TABLE_NAME}", "") assert len(files) == 1 create_hudi_table(instance, TABLE_NAME) assert int(instance.query(f"SELECT count() FROM {TABLE_NAME}")) == 100 write_hudi_from_df( spark, TABLE_NAME, generate_data(spark, 100, 200), f"/{TABLE_NAME}", mode="append", ) files = upload_directory(minio_client, bucket, f"/{TABLE_NAME}", "") assert len(files) == 2 assert int(instance.query(f"SELECT count() FROM {TABLE_NAME}")) == 200 assert instance.query( f"SELECT a, b FROM {TABLE_NAME} ORDER BY 1" ) == instance.query("SELECT number, toString(number + 1) FROM numbers(200)") write_hudi_from_df( spark, TABLE_NAME, generate_data(spark, 100, 300), f"/{TABLE_NAME}", mode="append", ) files = upload_directory(minio_client, bucket, f"/{TABLE_NAME}", "") assert len(files) == 3 assert int(instance.query(f"SELECT count() FROM {TABLE_NAME}")) == 300 assert instance.query( f"SELECT a, b FROM {TABLE_NAME} ORDER BY 1" ) == instance.query("SELECT number, toString(number + 1) FROM numbers(300)") assert int(instance.query(f"SELECT b FROM {TABLE_NAME} WHERE a = 100")) == 101 write_hudi_from_df( spark, TABLE_NAME, generate_data(spark, 100, 101, append=0), f"/{TABLE_NAME}", mode="append", ) files = upload_directory(minio_client, bucket, f"/{TABLE_NAME}", "") assert int(instance.query(f"SELECT count() FROM {TABLE_NAME}")) == 300 assert int(instance.query(f"SELECT b FROM {TABLE_NAME} WHERE a = 100")) == 100 > write_hudi_from_df( spark, TABLE_NAME, generate_data(spark, 100, 1000000, append=0), f"/{TABLE_NAME}", mode="append", ) test_storage_hudi/test.py:247: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_storage_hudi/test.py:110: in write_hudi_from_df ).save( /usr/local/lib/python3.10/dist-packages/pyspark/sql/readwriter.py:968: in save self._jwrite.save(path) /usr/local/lib/python3.10/dist-packages/py4j/java_gateway.py:1320: in __call__ answer = self.gateway_client.send_command(command) /usr/local/lib/python3.10/dist-packages/py4j/java_gateway.py:1038: in send_command response = connection.send_command(command) /usr/local/lib/python3.10/dist-packages/py4j/clientserver.py:511: in send_command answer = smart_decode(self.stream.readline()[:-1]) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = b = def readinto(self, b): """Read up to len(b) bytes into the writable buffer *b* and return the number of bytes read. If the socket is non-blocking and no bytes are available, None is returned. If *b* is non-empty, a 0 return value indicates that the connection was shutdown at the other end. """ self._checkClosed() self._checkReadable() if self._timeout_occurred: raise OSError("cannot read from timed out object") while True: try: > return self._sock.recv_into(b) E Failed: Timeout >900.0s /usr/lib/python3.10/socket.py:705: Failed ---------------------------- Captured stdout setup ----------------------------- 26/04/02 10:22:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ------------------------------ Captured log setup ------------------------------ 2026-04-02 10:21:56 [ 409 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 409 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-02 10:21:56 [ 409 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-04-02 10:21:56 [ 409 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-04-02 10:21:56 [ 409 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 409 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check) 2026-04-02 10:21:56 [ 409 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV HOSTNAME 5c0c783dd6f2 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV SHLVL 0 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV HOME /root (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV OLDPWD / (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_HELPER_TAG 2cffe1eae894 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke test_row_policy/test.py::test_some_users_without_policies test_row_policy/test.py::test_tags_with_db_and_table_names test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data test_row_policy/test.py::test_users_xml_is_readonly test_row_policy/test.py::test_with_prewhere test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_storage_hudi/test.py::test_multiple_hudi_files test_storage_hudi/test.py::test_single_hudi_file test_storage_hudi/test.py::test_types test_storage_redis/test.py::test_create_table test_storage_redis/test.py::test_delete test_storage_redis/test.py::test_select_int test_storage_redis/test.py::test_simple_insert test_storage_redis/test.py::test_simple_select test_storage_redis/test.py::test_truncate test_storage_redis/test.py::test_update test_storage_s3_queue/test.py::test_commit_on_limit 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered]' 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered]' -vvv (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_USE_OLD_ANALYZER 1 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV TZ Etc/UTC (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV DOCKER_BASE_TAG 1e0b53d756cf (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 0 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV WORKER_FREE_PORTS 30350 30351 30352 30353 30354 30355 30356 30357 30358 30359 30360 30361 30362 30363 30364 30365 30366 30367 30368 30369 30370 30371 30372 30373 30374 30375 30376 30377 30378 30379 30380 30381 30382 30383 30384 30385 30386 30387 30388 30389 30390 30391 30392 30393 30394 30395 30396 30397 30398 30399 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTEST_XDIST_TESTRUNUID 835aeb6757d841b8831c932b29425f1c (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTEST_XDIST_WORKER gw7 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : ENV PYTEST_CURRENT_TEST test_storage_hudi/test.py::test_multiple_hudi_files (setup) (cluster.py:446, __init__) 2026-04-02 10:21:56 [ 409 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:766, __init__) 2026-04-02 10:22:07 [ 409 ] DEBUG : GatewayClient.address is deprecated and will be removed in version 1.0. Use GatewayParameters instead. (java_gateway.py:163, deprecated) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: A 995e998689d0906077d0d8133c75418f7c8bc30f2dfdcbd279a6a94b87aec73e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.SparkConf e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.api.java.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.api.python.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.ml.python.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.mllib.api.python.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.resource.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.sql.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.sql.api.python.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj org.apache.spark.sql.hive.* e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: j i rj scala.Tuple2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: r u SparkConf rj e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.SparkConf (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: i org.apache.spark.SparkConf bTrue e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro0 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 set sspark.app.name sspark_test e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro1 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 set sspark.master slocal e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro2 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.serializer.objectStreamReset e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 set sspark.serializer.objectStreamReset s100 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro3 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.rdd.compress e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 set sspark.rdd.compress sTrue e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro4 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 get sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 get sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 contains sspark.home e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o0 getAll e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yto5 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i0 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro6 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o6 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.master (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o6 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro7 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o7 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.app.name (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o7 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o8 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.rdd.compress (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o8 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysTrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i3 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro9 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o9 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.serializer.objectStreamReset (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o9 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ys100 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i4 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro10 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o10 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.submit.pyFiles (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o10 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro11 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o11 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.submit.deployMode (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o11 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysclient (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i6 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro12 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o12 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.app.submitTime (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o12 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ys1775125327393 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a g o5 i7 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yro13 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o13 _1 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ysspark.ui.showConsoleProgress (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: c o13 _2 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ystrue (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: a e o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !yi8 (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: r u JavaSparkContext rj e (clientserver.py:501, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.java.JavaSparkContext (clientserver.py:512, send_command) 2026-04-02 10:22:07 [ 409 ] DEBUG : Command to send: i org.apache.spark.api.java.JavaSparkContext ro0 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: A 995e998689d0906077d0d8133c75418f7c8bc30f2dfdcbd279a6a94b87aec73e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: m d o1 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: m d o2 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: m d o3 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: m d o4 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Command to send: m d o5 e (clientserver.py:501, send_command) 2026-04-02 10:22:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro14 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro15 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o15 conf e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro16 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u PythonAccumulatorV2 rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonAccumulatorV2 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: i org.apache.spark.api.python.PythonAccumulatorV2 s127.0.0.1 i55915 s995e998689d0906077d0d8133c75418f7c8bc30f2dfdcbd279a6a94b87aec73e e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro17 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro18 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o18 register ro17 e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils isEncryptionEnabled e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils isEncryptionEnabled ro14 e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout ro14 e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yL15 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getSparkBufferSize e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getSparkBufferSize ro14 e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yi65536 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark.SparkFiles rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.SparkFiles (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7/userFiles-f70b665f-dc18-421b-ae16-3a1e6beebdc6 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o16 get sspark.submit.pyFiles s e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.util.Utils getLocalDir e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro19 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o19 conf e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro20 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils getLocalDir ro20 e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.util.Utils createTempDir e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils createTempDir s/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7 spyspark e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !yro21 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o21 getAbsolutePath e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7/pyspark-2e813497-342e-47af-bddd-d1f971c72ca5 (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: c o16 get sspark.python.profile sfalse e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:16 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yro22 (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c o22 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c o14 sc e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yro23 (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yao24 (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c o24 put sspark.app.name sspark_test e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c o24 put sspark.master slocal e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: i org.apache.spark.sql.SparkSession ro23 ro24 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yro25 (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setDefaultSession ro25 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setActiveSession ro25 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: c o14 stop e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o24 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o6 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o7 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o8 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o9 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o10 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o11 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o12 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o13 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o15 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o18 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o19 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o20 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o21 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o22 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Command to send: m d o23 e (clientserver.py:501, send_command) 2026-04-02 10:22:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession clearDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession clearDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession clearActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession clearActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:22:18 [ 409 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1785, add_instance) 2026-04-02 10:22:18 [ 409 ] DEBUG : Cluster name: project_name:rootteststoragehudi_gw7. Added instance name:node1 tag:1e0b53d756cf base_cmd:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:2079, add_instance) 2026-04-02 10:22:18 [ 409 ] INFO : Starting cluster... (test.py:58, started_cluster) 2026-04-02 10:22:18 [ 409 ] INFO : Running tests in /ClickHouse/tests/integration/test_storage_hudi/test.py (cluster.py:2788, start) 2026-04-02 10:22:18 [ 409 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start) 2026-04-02 10:22:18 [ 409 ] DEBUG : Docker networks for project rootteststoragehudi_gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:22:19 [ 409 ] DEBUG : Docker containers for project rootteststoragehudi_gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:22:19 [ 409 ] DEBUG : Docker volumes for project rootteststoragehudi_gw7 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:22:19 [ 409 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:22:19 [ 409 ] DEBUG : Docker networks for project rootteststoragehudi_gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:22:19 [ 409 ] DEBUG : Docker containers for project rootteststoragehudi_gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:22:20 [ 409 ] DEBUG : Docker volumes for project rootteststoragehudi_gw7 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:22:20 [ 409 ] DEBUG : Command:docker container list --all --filter name='^/rootteststoragehudi_gw7_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:22:20 [ 409 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : No running containers for project: rootteststoragehudi_gw7 (cluster.py:904, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:22:20 [ 409 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:22:20 [ 409 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:22:20 [ 409 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:22:21 [ 409 ] DEBUG : Stdout:2 (cluster.py:121, run_and_check) 2026-04-02 10:22:21 [ 409 ] DEBUG : Setup directory for instance: node1 (cluster.py:2808, start) 2026-04-02 10:22:21 [ 409 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_storage_hudi/configs/config.d/named_collections.xml'] to /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/database (cluster.py:4649, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/logs (cluster.py:4660, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-02 10:22:21 [ 409 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/certs/public.crt'} stored in /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env (cluster.py:86, _create_env_file) 2026-04-02 10:22:21 [ 409 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:22:21 [ 409 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:22:21 [ 409 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:22:21 [ 409 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:22:21 [ 409 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request) 2026-04-02 10:22:21 [ 409 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', 'pull'] (cluster.py:113, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy2 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling resolver ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling minio1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling node1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy2 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling resolver ... pulling from altinityinfra/python... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling minio1 ... pulling from minio/minio (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy1 ... pulling from altinityinfra/s3-proxy (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling node1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling resolver ... digest: sha256:919cb0e6c1a4f05e8d... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling resolver ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy2 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy2 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling resolver ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy1 ... digest: sha256:92e86c3de6b8ff00d8... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling minio1 ... digest: sha256:9eea83a4e1425067e7... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling minio1 ... status: image is up to date for m... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling minio1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling proxy1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling node1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling node1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] DEBUG : Stderr:Pulling node1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:34 [ 409 ] INFO : Trying to create Minio instance by command docker-compose --env-file /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env --project-name rootteststoragehudi_gw7 --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:3087, start) 2026-04-02 10:22:34 [ 409 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--verbose', 'up', '-d'] (cluster.py:113, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:docker-py version: (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-130-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.25', 'Details': {'GitCommit': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}}, {'Name': 'runc', 'Version': '1.2.4', 'Details': {'GitCommit': 'v1.2.4-0-g6c52b3f'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-130-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststoragehudigw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststoragehudigw7_data1-1') (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:123, run_and_check) 2026-04-02 10:22:55 [ 409 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'ID': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Containers': 30, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('rootteststoragehudi_gw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.network.ensure: Creating network "rootteststoragehudi_gw7_default" with the default driver (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='rootteststoragehudi_gw7_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'a0de6f93266c6b506b0872adcc9377bc69cea30543f4f23b67d354c6808905dc', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('rootteststoragehudi_gw7_data1-1') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.volume.initialize: Creating volume "rootteststoragehudi_gw7_data1-1" with default driver (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume <- ('rootteststoragehudi_gw7_data1-1', None, None, labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.volume': 'data1-1', 'com.docker.compose.version': '1.29.2'}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_volume -> {'CreatedAt': '2026-04-02T10:22:38Z', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Driver': 'local', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Labels': {'com.docker.compose.project': 'rootteststoragehudi_gw7', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'com.docker.compose.version': '1.29.2', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'com.docker.compose.volume': 'data1-1'}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Mountpoint': '/var/lib/docker/volumes/rootteststoragehudi_gw7_data1-1/_data', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Name': 'rootteststoragehudi_gw7_data1-1', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Options': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Scope': 'local'} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.project._get_convergence_plans: minio1 has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.project._get_convergence_plans: resolver has upstream changes (proxy1, proxy2) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'status': ['created', 'exited'], 'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , , } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_proxy1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststoragehudi_gw7', service='proxy1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststoragehudi_gw7', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=proxy2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_proxy2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststoragehudi_gw7', service='proxy2', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststoragehudi_gw7', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 3c576d479eff1810dcf0b45a8079e64c1bcf41412d7b6b2a7bd1058651945e30 (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststoragehudi_gw7_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NetworkMode': 'rootteststoragehudi_gw7_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststoragehudi_gw7_proxy1_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.service': 'proxy1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '3c576d479eff1810dcf0b45a8079e64c1bcf41412d7b6b2a7bd1058651945e30'}, host_config={'NetworkMode': 'rootteststoragehudi_gw7_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststoragehudi_gw7_default': {'Aliases': ['proxy1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/s3-proxy') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 3c576d479eff1810dcf0b45a8079e64c1bcf41412d7b6b2a7bd1058651945e30 (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststoragehudi_gw7_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NetworkMode': 'rootteststoragehudi_gw7_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/s3-proxy', volumes={}, name='rootteststoragehudi_gw7_proxy2_1', detach=True, ports=['8080', '80', '443'], environment=[], labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.service': 'proxy2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '3c576d479eff1810dcf0b45a8079e64c1bcf41412d7b6b2a7bd1058651945e30'}, host_config={'NetworkMode': 'rootteststoragehudi_gw7_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststoragehudi_gw7_default': {'Aliases': ['proxy2'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '1d00bb6645f044d3d1b45519cedf4bcb999eef73221b9b84a362fbf512f145af', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('1d00bb6645f044d3d1b45519cedf4bcb999eef73221b9b84a362fbf512f145af') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'dd68b550b0e3cea758d5780316509143d0e77b232200ea76d17767c3b5527a6e', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('dd68b550b0e3cea758d5780316509143d0e77b232200ea76d17767c3b5527a6e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('dd68b550b0e3cea758d5780316509143d0e77b232200ea76d17767c3b5527a6e', 'rootteststoragehudi_gw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Args': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['/run.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NGINX_VERSION=1.27.5', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('1d00bb6645f044d3d1b45519cedf4bcb999eef73221b9b84a362fbf512f145af', 'rootteststoragehudi_gw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('dd68b550b0e3cea758d5780316509143d0e77b232200ea76d17767c3b5527a6e', 'rootteststoragehudi_gw7_default', aliases=['dd68b550b0e3', 'proxy2'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('1d00bb6645f044d3d1b45519cedf4bcb999eef73221b9b84a362fbf512f145af', 'rootteststoragehudi_gw7_default', aliases=['proxy1', '1d00bb6645f0'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('dd68b550b0e3cea758d5780316509143d0e77b232200ea76d17767c3b5527a6e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('1d00bb6645f044d3d1b45519cedf4bcb999eef73221b9b84a362fbf512f145af') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststoragehudi_gw7', service='proxy2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststoragehudi_gw7', service='proxy1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, } (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=minio1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudi_gw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=rootteststoragehudigw7', 'com.docker.compose.service=resolver', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_minio1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststoragehudi_gw7', service='minio1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststoragehudi_gw7', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_resolver_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='rootteststoragehudi_gw7', service='resolver', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='rootteststoragehudi_gw7', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/python-bottle:caad4729259e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: b2ae2232c46ef9523be876aa5a1bfdd0be6da94f0911badc0a4aef8198c969a1 (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='rootteststoragehudi_gw7_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NetworkMode': 'rootteststoragehudi_gw7_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (image='altinityinfra/python-bottle:caad4729259e', tty=True, volumes={}, name='rootteststoragehudi_gw7_resolver_1', detach=True, ports=['8080'], environment=[], labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.service': 'resolver', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': 'b2ae2232c46ef9523be876aa5a1bfdd0be6da94f0911badc0a4aef8198c969a1'}, host_config={'NetworkMode': 'rootteststoragehudi_gw7_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststoragehudi_gw7_default': {'Aliases': ['resolver'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('minio/minio:RELEASE.2024-07-31T05-46-26Z') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['minio'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': ['/usr/bin/docker-entrypoint.sh'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 241af660d79aa4dd39335a6f74dcb203c173c9639d16f07482fb51e1bfb8b61b (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=['rootteststoragehudi_gw7_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/certs:/certs:rw'], volumes_from=[], privileged=False, network_mode='rootteststoragehudi_gw7_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': ['rootteststoragehudi_gw7_data1-1:/data1:rw', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/certs:/certs:rw'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'NetworkMode': 'rootteststoragehudi_gw7_default', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'PortBindings': {}, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'VolumesFrom': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='server --console-address 127.0.0.1:19001 --address :9001 --certs-dir /certs /data1-1', environment=['MINIO_ACCESS_KEY=minio', 'MINIO_SECRET_KEY=minio123', 'MINIO_PROMETHEUS_AUTH_TYPE=public'], image='minio/minio:RELEASE.2024-07-31T05-46-26Z', volumes={'/data1': {}, '/certs': {}}, name='rootteststoragehudi_gw7_minio1_1', detach=True, ports=['9001'], labels={'com.docker.compose.project': 'rootteststoragehudi_gw7', 'com.docker.compose.service': 'minio1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_minio.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '241af660d79aa4dd39335a6f74dcb203c173c9639d16f07482fb51e1bfb8b61b'}, host_config={'NetworkMode': 'rootteststoragehudi_gw7_default', 'VolumesFrom': [], 'Binds': ['rootteststoragehudi_gw7_data1-1:/data1:rw', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/minio/certs:/certs:rw'], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'rootteststoragehudi_gw7_default': {'Aliases': ['minio1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '5bbec463a1170cf141c90190cb49f0c5a6b251bfe76da8e5dd9dd931e5fcef2e', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('5bbec463a1170cf141c90190cb49f0c5a6b251bfe76da8e5dd9dd931e5fcef2e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Args': [], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Cmd': ['python3'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Env': ['PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '5593f5f3a52c3569d1dafd7673e8de7420797da772492c5a21fb3301d533897d', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('5593f5f3a52c3569d1dafd7673e8de7420797da772492c5a21fb3301d533897d') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('5bbec463a1170cf141c90190cb49f0c5a6b251bfe76da8e5dd9dd931e5fcef2e', 'rootteststoragehudi_gw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Args': ['server', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '--console-address', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '127.0.0.1:19001', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '--address', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: ':9001', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '--certs-dir', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '/certs', (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: '/data1-1'], (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('5593f5f3a52c3569d1dafd7673e8de7420797da772492c5a21fb3301d533897d', 'rootteststoragehudi_gw7_default') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('5bbec463a1170cf141c90190cb49f0c5a6b251bfe76da8e5dd9dd931e5fcef2e', 'rootteststoragehudi_gw7_default', aliases=['resolver', '5bbec463a117'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('5593f5f3a52c3569d1dafd7673e8de7420797da772492c5a21fb3301d533897d', 'rootteststoragehudi_gw7_default', aliases=['minio1', '5593f5f3a52c'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('5bbec463a1170cf141c90190cb49f0c5a6b251bfe76da8e5dd9dd931e5fcef2e') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('5593f5f3a52c3569d1dafd7673e8de7420797da772492c5a21fb3301d533897d') (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststoragehudi_gw7', service='resolver', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_resolver_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='rootteststoragehudi_gw7', service='minio1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_minio1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:22:56 [ 409 ] INFO : Trying to connect to Minio... (cluster.py:3093, start) 2026-04-02 10:22:56 [ 409 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:56 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragehudi_gw7_minio1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:56 [ 409 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:22:56 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragehudi_gw7_proxy1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:22:56 [ 409 ] DEBUG : Starting new HTTP connection (1): 172.16.10.5:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:56 [ 409 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:22:56 [ 409 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:22:56 [ 409 ] DEBUG : Starting new HTTP connection (2): 172.16.10.5:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:56 [ 409 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:22:56 [ 409 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:22:56 [ 409 ] DEBUG : Starting new HTTP connection (3): 172.16.10.5:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:56 [ 409 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2026-04-02 10:22:56 [ 409 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2026-04-02 10:22:56 [ 409 ] DEBUG : Starting new HTTP connection (4): 172.16.10.5:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:56 [ 409 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.10.5', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2669, wait_minio_to_start) 2026-04-02 10:22:57 [ 409 ] DEBUG : Starting new HTTP connection (5): 172.16.10.5:9001 (connectionpool.py:245, _new_conn) 2026-04-02 10:22:57 [ 409 ] DEBUG : http://172.16.10.5:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:22:57 [ 409 ] DEBUG : Connected to Minio. (cluster.py:2649, wait_minio_to_start) 2026-04-02 10:22:57 [ 409 ] DEBUG : http://172.16.10.5:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2026-04-02 10:22:57 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:22:57 [ 409 ] DEBUG : S3 bucket 'root' created (cluster.py:2664, wait_minio_to_start) 2026-04-02 10:22:57 [ 409 ] DEBUG : http://172.16.10.5:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2026-04-02 10:22:57 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:22:57 [ 409 ] DEBUG : S3 bucket 'root2' created (cluster.py:2664, wait_minio_to_start) 2026-04-02 10:22:57 [ 409 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env --project-name rootteststoragehudi_gw7 --file /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml up -d --no-recreate') (cluster.py:3146, start) 2026-04-02 10:22:57 [ 409 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-04-02 10:23:09 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:23:09 [ 409 ] DEBUG : Stderr:Creating rootteststoragehudi_gw7_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:23:09 [ 409 ] DEBUG : ClickHouse instance created (cluster.py:3154, start) 2026-04-02 10:23:09 [ 409 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:23:09 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragehudi_gw7_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:09 [ 409 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.10.6... (cluster.py:3161, start) 2026-04-02 10:23:09 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/rootteststoragehudi_gw7_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:09 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:09 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:10 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:11 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:12 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:13 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:14 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:15 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:16 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:17 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:18 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:19 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:20 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:21 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://localhost:None "GET /v1.42/containers/2ad55ad4eb1a8dd8d551ec12666118f571c8f8f92a53adcf0eb20d0a7884f463/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : ClickHouse node1 started (cluster.py:3165, start) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root?policy= HTTP/1.1" 204 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://172.16.10.5:9001 "GET /root-with-auth?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root-with-auth HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:23:22 [ 409 ] INFO : S3 bucket created (test.py:62, started_cluster) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: r u SparkConf rj e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.SparkConf (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: i org.apache.spark.SparkConf bTrue e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro26 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.app.name sspark_test e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro27 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.master slocal e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro28 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sorg.apache.spark.sql.hudi.catalog.HoodieCatalog sNone e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro29 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.serializer sorg.apache.spark.serializer.KryoSerializer e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro30 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.sql.catalog.local sorg.apache.spark.sql.hudi.catalog.HoodieCatalog e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro31 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.driver.memory s20g e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !yro32 (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 get sspark.executor.allowSparkContext sfalse e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 contains sspark.serializer.objectStreamReset e (clientserver.py:501, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:23:22 [ 409 ] DEBUG : Command to send: c o26 set sspark.serializer.objectStreamReset s100 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro33 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.rdd.compress e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 set sspark.rdd.compress sTrue e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro34 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 get sspark.master e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 get sspark.app.name e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 contains sspark.home e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o26 getAll e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yto35 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i0 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro36 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o36 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.master (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o36 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yslocal (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro37 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o37 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.serializer (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o37 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysorg.apache.spark.serializer.KryoSerializer (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro38 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o38 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.app.name (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o38 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark_test (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i3 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro39 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o39 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.driver.memory (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o39 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ys20g (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i4 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro40 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o40 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.rdd.compress (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o40 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysTrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i5 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro41 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o41 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.serializer.objectStreamReset (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o41 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ys100 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i6 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro42 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o42 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.submit.pyFiles (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o42 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i7 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro43 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o43 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysorg.apache.spark.sql.hudi.catalog.HoodieCatalog (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o43 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysNone (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i8 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro44 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o44 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.submit.deployMode (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o44 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysclient (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i9 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro45 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o45 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.app.submitTime (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o45 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ys1775125327393 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i10 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro46 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o46 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.ui.showConsoleProgress (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o46 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ystrue (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a g o35 i11 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yro47 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o47 _1 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysspark.sql.catalog.local (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: c o47 _2 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ysorg.apache.spark.sql.hudi.catalog.HoodieCatalog (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: a e o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yi12 (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: r u JavaSparkContext rj e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.java.JavaSparkContext (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: i org.apache.spark.api.java.JavaSparkContext ro26 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o27 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o28 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o29 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o30 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o31 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o32 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o33 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o34 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o36 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o37 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o38 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o39 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o40 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o41 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o42 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o43 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o44 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o45 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Command to send: m d o35 e (clientserver.py:501, send_command) 2026-04-02 10:23:23 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yro48 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yro49 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c o49 conf e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yro50 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u PythonAccumulatorV2 rj e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonAccumulatorV2 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: i org.apache.spark.api.python.PythonAccumulatorV2 s127.0.0.1 i59119 s995e998689d0906077d0d8133c75418f7c8bc30f2dfdcbd279a6a94b87aec73e e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yro51 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yro52 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c o52 register ro51 e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils isEncryptionEnabled e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils isEncryptionEnabled ro48 e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout ro48 e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yL15 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils getSparkBufferSize e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils getSparkBufferSize ro48 e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yi65536 (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:24 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark.SparkFiles rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.SparkFiles (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r m org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.SparkFiles getRootDirectory e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7/userFiles-2c59c952-b397-471c-95a1-55890afa50a4 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o50 get sspark.submit.pyFiles s e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ys (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r m org.apache.spark.util.Utils getLocalDir e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro53 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o53 conf e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro54 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils getLocalDir ro54 e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u org.apache.spark.util.Utils rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.util.Utils (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r m org.apache.spark.util.Utils createTempDir e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.util.Utils createTempDir s/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7 spyspark e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro55 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o55 getAbsolutePath e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ys/tmp/spark-be76a097-ec5c-4b5d-9e13-4ef36222cae7/pyspark-10046537-1486-46f6-9c27-71f3dd45916d (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o50 get sspark.python.profile sfalse e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ysfalse (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro56 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o56 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ybfalse (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro57 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yao58 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sspark.app.name sspark_test e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sspark.master slocal e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sorg.apache.spark.sql.hudi.catalog.HoodieCatalog sNone e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sspark.serializer sorg.apache.spark.serializer.KryoSerializer e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sspark.sql.catalog.local sorg.apache.spark.sql.hudi.catalog.HoodieCatalog e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c o58 put sspark.driver.memory s20g e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yn (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: i org.apache.spark.sql.SparkSession ro57 ro58 e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !yro59 (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setDefaultSession e (clientserver.py:501, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:25 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setDefaultSession ro59 e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession setActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession setActiveSession ro59 e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) ----------------------------- Captured stdout call ----------------------------- 26/04/02 10:24:25 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf 26/04/02 10:24:25 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file 26/04/02 10:24:44 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:24:44 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:24:44 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:24:44 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:25:07 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:25:07 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:25:10 WARN HoodieBackedTableMetadata: Metadata table was not found at path /test_multiple_hudi_files/.hoodie/metadata 26/04/02 10:25:13 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark # WARNING: Unable to attach Serviceability Agent. You can try again with escalated privileges. Two options: a) use -Djol.tryWithSudo=true to try with sudo; b) echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope 26/04/02 10:26:51 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark 26/04/02 10:27:05 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet 26/04/02 10:27:31 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:31 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:31 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:31 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:33 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:33 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:27:34 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark 26/04/02 10:27:53 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet 26/04/02 10:28:20 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:20 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:20 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:20 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:24 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:24 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:28:25 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark 26/04/02 10:28:40 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet 26/04/02 10:29:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:14 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:15 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:15 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:16 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark 26/04/02 10:29:27 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet Putting file /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-107-83_20260402102912282.parquet to /test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-107-83_20260402102912282.parquet 26/04/02 10:29:50 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:50 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:50 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:29:50 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:30:07 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:30:07 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation. 26/04/02 10:30:08 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark ----------------------------- Captured stderr call ----------------------------- [Stage 0:> (0 + 0) / 1] [Stage 0:> (0 + 1) / 1] [Stage 0:===========================================================(1 + 0) / 1] [Stage 1:> (0 + 1) / 1] [Stage 2:> (0 + 1) / 1] [Stage 2:===========================================================(1 + 0) / 1] [Stage 3:> (0 + 1) / 1] [Stage 4:> (0 + 0) / 1] [Stage 4:> (0 + 1) / 1] [Stage 5:> (0 + 1) / 1] [Stage 6:> (0 + 0) / 1] [Stage 6:> (0 + 1) / 1] [Stage 8:> (0 + 0) / 1] [Stage 8:> (0 + 1) / 1] [Stage 10:> (0 + 0) / 1] [Stage 10:> (0 + 1) / 1] [Stage 11:> (0 + 1) / 1] [Stage 13:> (0 + 0) / 1] [Stage 13:> (0 + 1) / 1] [Stage 14:==========================================================(1 + 0) / 1] [Stage 15:> (0 + 1) / 1] [Stage 21:> (0 + 1) / 1] [Stage 24:> (0 + 0) / 1] [Stage 24:> (0 + 1) / 1] [Stage 25:> (0 + 1) / 1] [Stage 26:> (0 + 0) / 1] [Stage 27:> (0 + 1) / 1] [Stage 29:> (0 + 1) / 1] [Stage 29:> (0 + 1) / 1][Stage 30:> (0 + 0) / 1] [Stage 30:> (0 + 1) / 1] [Stage 30:==========================================================(1 + 0) / 1] [Stage 31:> (0 + 1) / 1] [Stage 33:> (0 + 0) / 1] [Stage 33:> (0 + 1) / 1] [Stage 38:> (0 + 1) / 1] [Stage 43:> (0 + 0) / 1] [Stage 43:> (0 + 1) / 1] [Stage 46:> (0 + 1) / 1] [Stage 47:> (0 + 0) / 1] [Stage 47:> (0 + 1) / 1] [Stage 48:> (0 + 1) / 1] [Stage 54:> (0 + 1) / 1] [Stage 56:> (0 + 1) / 1] [Stage 60:> (0 + 0) / 1] [Stage 60:> (0 + 1) / 1] [Stage 62:> (0 + 1) / 1][Stage 63:> (0 + 0) / 1] [Stage 63:> (0 + 1) / 1] [Stage 64:> (0 + 1) / 1] [Stage 66:> (0 + 1) / 1] [Stage 70:> (0 + 0) / 1] [Stage 70:> (0 + 1) / 1] [Stage 71:> (0 + 1) / 1] [Stage 80:> (0 + 1) / 1] [Stage 81:> (0 + 1) / 1] [Stage 81:==========================================================(1 + 0) / 1] [Stage 87:> (0 + 1) / 1] [Stage 96:> (0 + 1) / 1] [Stage 106:> (0 + 1) / 1] [Stage 107:> (0 + 1) / 1] [Stage 117:> (0 + 1) / 1] [Stage 128:> (0 + 1) / 1] [Stage 129:> (0 + 1) / 1] [Stage 130:> (0 + 0) / 1] [Stage 130:> (0 + 1) / 1] [Stage 130:> (0 + 1) / 1] [Stage 132:> (0 + 1) / 1] [Stage 134:> (0 + 0) / 1] [Stage 134:> (0 + 1) / 1][Stage 135:> (0 + 0) / 1] [Stage 134:> (0 + 1) / 1][Stage 135:> (0 + 0) / 1] [Stage 134:> (0 + 1) / 1][Stage 135:> (0 + 0) / 1] [Stage 135:> (0 + 1) / 1] [Stage 136:> (0 + 1) / 1] [Stage 136:> (0 + 1) / 1] [Stage 138:> (0 + 1) / 1] [Stage 142:> (0 + 1) / 1] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++ ~~~~~~~~~~~~~ Stack of Thread-3 (serve_forever) (139952553666112) ~~~~~~~~~~~~~~ File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.10/socketserver.py", line 232, in serve_forever ready = selector.select(poll_interval) File "/usr/lib/python3.10/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) ~~~~~~~~~~~~~~~~~~~~~ Stack of Thread-1 (139952545273408) ~~~~~~~~~~~~~~~~~~~~~~ File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/lib/python3.10/dist-packages/py4j/clientserver.py", line 58, in run time.sleep(DEFAULT_WORKER_SLEEP_TIME) ~~~~~~~~~~~~~~~~~~~~~ Stack of (139953343632960) ~~~~~~~~~~~~~~~~~~~~~ File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 411, in _perform_spawn reply.run() File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 341, in run self._result = func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 1160, in _thread_receiver msg = Message.from_io(io) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 567, in from_io header = io.read(9) # type 1, channel 4, payload 4 File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 534, in read data = self._read(numbytes - len(buf)) +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++ ------------------------------ Captured log call ------------------------------- 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !yro60 (clientserver.py:512, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: c o60 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: c o59 range i0 i100 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Command to send: m d o58 e (clientserver.py:501, send_command) 2026-04-02 10:23:26 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !yro61 (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !ylo62 (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: c o62 add sa e (clientserver.py:501, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro62 e (clientserver.py:501, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Answer received: !yro63 (clientserver.py:512, send_command) 2026-04-02 10:24:07 [ 409 ] DEBUG : Command to send: c o61 toDF ro63 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: m d o62 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro64 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro65 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o65 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o59 range i1 i101 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro66 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ylo67 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o67 add sb e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro67 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro68 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o66 toDF ro68 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro69 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o69 apply sb e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro70 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro71 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o71 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro72 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o72 get e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro73 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro74 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yao75 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o74 applyModifiableSettings ro73 ro75 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o59 parseDataType s"string" e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro76 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o70 cast ro76 e (clientserver.py:501, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Answer received: !yro77 (clientserver.py:512, send_command) 2026-04-02 10:24:08 [ 409 ] DEBUG : Command to send: c o69 withColumn sb ro77 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro78 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o67 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o47 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o49 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o52 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o53 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o54 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o55 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o56 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o57 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o60 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o61 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o63 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o65 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o66 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o68 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: m d o75 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro79 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro80 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ylo81 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c o81 add ro80 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro81 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro82 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro82 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro83 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c o79 over ro83 e (clientserver.py:501, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Answer received: !yro84 (clientserver.py:512, send_command) 2026-04-02 10:24:09 [ 409 ] DEBUG : Command to send: c o64 withColumn srow_index ro84 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro85 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro86 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro87 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ylo88 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c o88 add ro87 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro88 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro89 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro89 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro90 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c o86 over ro90 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro91 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c o78 withColumn srow_index ro91 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: m d o81 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: m d o88 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro92 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ylo93 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c o93 add srow_index e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro93 e (clientserver.py:501, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Answer received: !yro94 (clientserver.py:512, send_command) 2026-04-02 10:24:10 [ 409 ] DEBUG : Command to send: c o85 join ro92 ro94 sinner e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: m d o93 e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro95 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o95 drop srow_index e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro96 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o96 write e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro97 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o97 mode soverwrite e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro98 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o98 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro99 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o99 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro100 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o100 format shudi e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro101 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o101 option shoodie.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro102 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o102 option shoodie.datasource.write.partitionpath.field spartitionpath e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro103 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o103 option shoodie.datasource.write.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro104 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o104 option shoodie.datasource.write.operation sinsert_overwrite e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro105 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o105 option shoodie.datasource.compaction.async.enable strue e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro106 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o106 option shoodie.compact.inline sfalse e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro107 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o107 option shoodie.compact.inline.max.delta.commits s10 e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro108 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o108 option shoodie.parquet.compression.codec ssnappy e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro109 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o109 option shoodie.hfile.compression.algorithm suncompressed e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro110 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o110 option shoodie.datasource.write.recordkey.field sa e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro111 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o111 option shoodie.datasource.write.precombine.field sa e (clientserver.py:501, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Answer received: !yro112 (clientserver.py:512, send_command) 2026-04-02 10:24:11 [ 409 ] DEBUG : Command to send: c o112 save s/test_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o70 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o71 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o72 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o74 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o76 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o77 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o78 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o79 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o80 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o82 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o83 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o84 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o85 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o86 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o87 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o89 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o90 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o91 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o92 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o94 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o95 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o97 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o98 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o99 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o100 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o101 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o102 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o103 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o104 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o105 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o106 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Command to send: m d o107 e (clientserver.py:501, send_command) 2026-04-02 10:24:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:17 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:17 [ 409 ] DEBUG : Resetting dropped connection: 172.16.10.5 (connectionpool.py:293, _get_conn) 2026-04-02 10:27:17 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:27:17 [ 409 ] DEBUG : Executing query DROP TABLE IF EXISTS test_multiple_hudi_files; CREATE TABLE test_multiple_hudi_files ENGINE=Hudi(s3, filename = 'test_multiple_hudi_files/') on node1 (cluster.py:3602, query) 2026-04-02 10:27:23 [ 409 ] DEBUG : Executing query SELECT count() FROM test_multiple_hudi_files on node1 (cluster.py:3602, query) 2026-04-02 10:27:26 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:27:26 [ 409 ] DEBUG : Answer received: !yro113 (clientserver.py:512, send_command) 2026-04-02 10:27:26 [ 409 ] DEBUG : Command to send: c o113 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:27:26 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:27:26 [ 409 ] DEBUG : Command to send: c o59 range i100 i200 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro114 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ylo115 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o115 add sa e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro115 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro116 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o114 toDF ro116 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro117 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro118 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o118 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o59 range i101 i201 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro119 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ylo120 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o120 add sb e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro120 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro121 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o119 toDF ro121 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro122 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o122 apply sb e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro123 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro124 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o124 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro125 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o125 get e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro126 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro127 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yao128 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o127 applyModifiableSettings ro126 ro128 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o59 parseDataType s"string" e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro129 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o123 cast ro129 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro130 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o122 withColumn sb ro130 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro131 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro132 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro133 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ylo134 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o134 add ro133 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro134 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro135 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro135 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro136 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o132 over ro136 e (clientserver.py:501, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Answer received: !yro137 (clientserver.py:512, send_command) 2026-04-02 10:27:27 [ 409 ] DEBUG : Command to send: c o117 withColumn srow_index ro137 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro138 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro139 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro140 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ylo141 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o141 add ro140 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro141 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro142 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro142 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro143 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o139 over ro143 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro144 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o131 withColumn srow_index ro144 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o115 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o120 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o128 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o134 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o110 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o111 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o112 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o113 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o114 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o116 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o117 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o118 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o119 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o121 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o122 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o123 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o124 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o125 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o127 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o129 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o130 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o132 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o133 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o135 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o136 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o137 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: m d o141 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro145 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ylo146 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o146 add srow_index e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro146 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro147 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o138 join ro145 ro147 sinner e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro148 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o148 drop srow_index e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro149 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o149 write e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro150 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o150 mode sappend e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro151 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o151 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro152 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o152 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro153 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o153 format shudi e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro154 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o154 option shoodie.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro155 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o155 option shoodie.datasource.write.partitionpath.field spartitionpath e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro156 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o156 option shoodie.datasource.write.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro157 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o157 option shoodie.datasource.write.operation supsert e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro158 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o158 option shoodie.datasource.compaction.async.enable strue e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro159 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o159 option shoodie.compact.inline sfalse e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro160 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o160 option shoodie.compact.inline.max.delta.commits s10 e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro161 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o161 option shoodie.parquet.compression.codec ssnappy e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro162 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o162 option shoodie.hfile.compression.algorithm suncompressed e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro163 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o163 option shoodie.datasource.write.recordkey.field sa e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro164 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o164 option shoodie.datasource.write.precombine.field sa e (clientserver.py:501, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Answer received: !yro165 (clientserver.py:512, send_command) 2026-04-02 10:27:28 [ 409 ] DEBUG : Command to send: c o165 save s/test_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:27:29 [ 409 ] DEBUG : Command to send: m d o146 e (clientserver.py:501, send_command) 2026-04-02 10:27:29 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:06 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:06 [ 409 ] DEBUG : Resetting dropped connection: 172.16.10.5 (connectionpool.py:293, _get_conn) 2026-04-02 10:28:06 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:28:06 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:28:06 [ 409 ] DEBUG : Executing query SELECT count() FROM test_multiple_hudi_files on node1 (cluster.py:3602, query) 2026-04-02 10:28:11 [ 409 ] DEBUG : Executing query SELECT a, b FROM test_multiple_hudi_files ORDER BY 1 on node1 (cluster.py:3602, query) 2026-04-02 10:28:14 [ 409 ] DEBUG : Executing query SELECT number, toString(number + 1) FROM numbers(200) on node1 (cluster.py:3602, query) 2026-04-02 10:28:17 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:28:17 [ 409 ] DEBUG : Answer received: !yro166 (clientserver.py:512, send_command) 2026-04-02 10:28:17 [ 409 ] DEBUG : Command to send: c o166 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:28:17 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:28:17 [ 409 ] DEBUG : Command to send: c o59 range i100 i300 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro167 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ylo168 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o168 add sa e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro168 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro169 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o167 toDF ro169 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro170 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro171 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o171 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o59 range i101 i301 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro172 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ylo173 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o173 add sb e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro173 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro174 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o172 toDF ro174 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro175 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o175 apply sb e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro176 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro177 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o177 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro178 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o178 get e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro179 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro180 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yao181 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o180 applyModifiableSettings ro179 ro181 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o59 parseDataType s"string" e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro182 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o176 cast ro182 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro183 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o175 withColumn sb ro183 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro184 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro185 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro186 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o168 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o173 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o181 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o0 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o14 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o17 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o25 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o16 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o46 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o26 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o64 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ylo187 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o187 add ro186 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o69 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o108 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o109 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro187 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o96 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o73 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o131 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o138 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o140 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o139 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o126 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro188 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro188 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro189 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o185 over ro189 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o142 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yro190 (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: c o170 withColumn srow_index ro190 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o143 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o144 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o145 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o147 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o148 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o149 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o150 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o151 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o152 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o153 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o154 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o155 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o156 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o157 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o158 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o159 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o160 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o161 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o162 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o163 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o164 e (clientserver.py:501, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:18 [ 409 ] DEBUG : Command to send: m d o165 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o166 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o167 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o169 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o171 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o172 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o174 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o175 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o176 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o177 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o178 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o180 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o182 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o183 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: m d o187 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro191 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro192 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro193 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ylo194 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o194 add ro193 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro194 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro195 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro195 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro196 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o192 over ro196 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro197 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o184 withColumn srow_index ro197 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro198 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ylo199 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o199 add srow_index e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro199 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro200 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o191 join ro198 ro200 sinner e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro201 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o201 drop srow_index e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro202 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o202 write e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro203 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o203 mode sappend e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro204 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o204 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro205 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o205 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro206 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o206 format shudi e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro207 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o207 option shoodie.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro208 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o208 option shoodie.datasource.write.partitionpath.field spartitionpath e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro209 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o209 option shoodie.datasource.write.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro210 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o210 option shoodie.datasource.write.operation supsert e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro211 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o211 option shoodie.datasource.compaction.async.enable strue e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro212 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o212 option shoodie.compact.inline sfalse e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro213 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o213 option shoodie.compact.inline.max.delta.commits s10 e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro214 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o214 option shoodie.parquet.compression.codec ssnappy e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro215 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o215 option shoodie.hfile.compression.algorithm suncompressed e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro216 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o216 option shoodie.datasource.write.recordkey.field sa e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro217 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o217 option shoodie.datasource.write.precombine.field sa e (clientserver.py:501, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Answer received: !yro218 (clientserver.py:512, send_command) 2026-04-02 10:28:19 [ 409 ] DEBUG : Command to send: c o218 save s/test_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:28:20 [ 409 ] DEBUG : Command to send: m d o194 e (clientserver.py:501, send_command) 2026-04-02 10:28:20 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:20 [ 409 ] DEBUG : Command to send: m d o199 e (clientserver.py:501, send_command) 2026-04-02 10:28:20 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:58 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:58 [ 409 ] DEBUG : Resetting dropped connection: 172.16.10.5 (connectionpool.py:293, _get_conn) 2026-04-02 10:28:58 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:28:58 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:28:58 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:28:58 [ 409 ] DEBUG : Executing query SELECT count() FROM test_multiple_hudi_files on node1 (cluster.py:3602, query) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o188 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o189 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o190 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o191 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o192 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o193 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o195 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o196 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o197 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o198 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o200 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o201 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o202 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o203 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o204 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o205 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o206 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o207 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o208 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o209 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o210 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o211 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o212 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o213 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o214 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o215 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o216 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o217 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Command to send: m d o218 e (clientserver.py:501, send_command) 2026-04-02 10:28:59 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:01 [ 409 ] DEBUG : Executing query SELECT a, b FROM test_multiple_hudi_files ORDER BY 1 on node1 (cluster.py:3602, query) 2026-04-02 10:29:04 [ 409 ] DEBUG : Executing query SELECT number, toString(number + 1) FROM numbers(300) on node1 (cluster.py:3602, query) 2026-04-02 10:29:07 [ 409 ] DEBUG : Executing query SELECT b FROM test_multiple_hudi_files WHERE a = 100 on node1 (cluster.py:3602, query) 2026-04-02 10:29:10 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:29:10 [ 409 ] DEBUG : Answer received: !yro219 (clientserver.py:512, send_command) 2026-04-02 10:29:10 [ 409 ] DEBUG : Command to send: c o219 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:29:10 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:29:10 [ 409 ] DEBUG : Command to send: c o59 range i100 i101 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro220 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ylo221 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o221 add sa e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro221 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro222 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o220 toDF ro222 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro223 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro224 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o224 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o59 range i100 i101 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro225 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ylo226 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o226 add sb e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro226 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro227 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o225 toDF ro227 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro228 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o228 apply sb e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro229 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro230 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o230 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro231 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o231 get e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro232 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro233 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yao234 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o233 applyModifiableSettings ro232 ro234 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o59 parseDataType s"string" e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro235 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o229 cast ro235 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro236 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o228 withColumn sb ro236 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro237 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro238 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro239 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ylo240 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o240 add ro239 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro240 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro241 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro241 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro242 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o238 over ro242 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro243 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o223 withColumn srow_index ro243 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro244 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro245 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro246 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: m d o221 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: m d o226 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: m d o234 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: m d o240 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ylo247 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o247 add ro246 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro247 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro248 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro248 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro249 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o245 over ro249 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro250 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o237 withColumn srow_index ro250 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro251 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ylo252 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o252 add srow_index e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro252 e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro253 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o244 join ro251 ro253 sinner e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro254 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o254 drop srow_index e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro255 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o255 write e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro256 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o256 mode sappend e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro257 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o257 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro258 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o258 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro259 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o259 format shudi e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro260 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o260 option shoodie.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro261 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o261 option shoodie.datasource.write.partitionpath.field spartitionpath e (clientserver.py:501, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Answer received: !yro262 (clientserver.py:512, send_command) 2026-04-02 10:29:11 [ 409 ] DEBUG : Command to send: c o262 option shoodie.datasource.write.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro263 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o263 option shoodie.datasource.write.operation supsert e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro264 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o264 option shoodie.datasource.compaction.async.enable strue e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro265 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o265 option shoodie.compact.inline sfalse e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro266 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o266 option shoodie.compact.inline.max.delta.commits s10 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro267 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o267 option shoodie.parquet.compression.codec ssnappy e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro268 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o268 option shoodie.hfile.compression.algorithm suncompressed e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro269 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o269 option shoodie.datasource.write.recordkey.field sa e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro270 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o270 option shoodie.datasource.write.precombine.field sa e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yro271 (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: c o271 save s/test_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o247 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o252 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o219 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o220 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o222 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o223 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o224 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o225 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o227 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o228 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o229 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o230 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o231 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o233 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o235 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o236 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o237 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o238 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o239 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o241 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o242 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o243 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o245 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o246 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o248 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o249 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o250 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Command to send: m d o253 e (clientserver.py:501, send_command) 2026-04-02 10:29:12 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:38 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:38 [ 409 ] DEBUG : Resetting dropped connection: 172.16.10.5 (connectionpool.py:293, _get_conn) 2026-04-02 10:29:38 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-38-31_20260402102728683.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:29:38 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-5-5_20260402102425976.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:29:39 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-71-55_20260402102819527.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:29:39 [ 409 ] DEBUG : http://172.16.10.5:9001 "PUT /root//test_multiple_hudi_files/__HIVE_DEFAULT_PARTITION__/f0c81e24-0f16-4b7c-bafc-d304f0d3e7c9-0_0-107-83_20260402102912282.parquet HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2026-04-02 10:29:39 [ 409 ] DEBUG : Executing query SELECT count() FROM test_multiple_hudi_files on node1 (cluster.py:3602, query) 2026-04-02 10:29:46 [ 409 ] DEBUG : Executing query SELECT b FROM test_multiple_hudi_files WHERE a = 100 on node1 (cluster.py:3602, query) 2026-04-02 10:29:48 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Answer received: !yro272 (clientserver.py:512, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Command to send: c o272 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Command to send: c o59 range i100 i1000000 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Answer received: !yro273 (clientserver.py:512, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:48 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ylo274 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o274 add sa e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro274 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro275 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o273 toDF ro275 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro276 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o48 sc e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro277 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o277 defaultParallelism e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yi1 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o59 range i100 i1000000 i1 i1 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro278 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ylo279 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o279 add sb e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro279 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro280 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o278 toDF ro280 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro281 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o281 apply sb e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro282 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro283 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o283 isDefined e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.SparkSession getActiveSession e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro284 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o284 get e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro285 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u SparkSession$ rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.SparkSession$ (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.SparkSession$ MODULE$ e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro286 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.HashMap e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yao287 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o286 applyModifiableSettings ro285 ro287 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o59 parseDataType s"string" e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro288 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o282 cast ro288 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro289 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o281 withColumn sb ro289 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro290 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro291 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro292 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ylo293 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o293 add ro292 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro293 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro294 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro294 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro295 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o291 over ro295 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro296 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o276 withColumn srow_index ro296 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro297 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions row_number e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro298 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u functions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.functions (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.functions monotonically_increasing_id e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro299 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yp (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u org.apache.spark.sql.expressions.Window rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.sql.expressions.Window (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.sql.expressions.Window orderBy e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ylo300 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o300 add ro299 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro300 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro301 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.sql.expressions.Window orderBy ro301 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro302 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o298 over ro302 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro303 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o290 withColumn srow_index ro303 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro304 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r u PythonUtils rj e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ycorg.apache.spark.api.python.PythonUtils (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: r m org.apache.spark.api.python.PythonUtils toSeq e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ym (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: i java.util.ArrayList e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ylo305 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o305 add srow_index e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !ybtrue (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c z:org.apache.spark.api.python.PythonUtils toSeq ro305 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro306 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o297 join ro304 ro306 sinner e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro307 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o307 drop srow_index e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro308 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o308 write e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro309 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o309 mode sappend e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro310 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o310 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro311 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o311 option scompression snone e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro312 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o312 format shudi e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro313 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o313 option shoodie.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro314 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o314 option shoodie.datasource.write.partitionpath.field spartitionpath e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro315 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o315 option shoodie.datasource.write.table.name stest_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro316 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o316 option shoodie.datasource.write.operation supsert e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro317 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o317 option shoodie.datasource.compaction.async.enable strue e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro318 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o318 option shoodie.compact.inline sfalse e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro319 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o319 option shoodie.compact.inline.max.delta.commits s10 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro320 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o320 option shoodie.parquet.compression.codec ssnappy e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro321 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o321 option shoodie.hfile.compression.algorithm suncompressed e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro322 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o322 option shoodie.datasource.write.recordkey.field sa e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro323 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o323 option shoodie.datasource.write.precombine.field sa e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yro324 (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: c o324 save s/test_multiple_hudi_files e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o274 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o279 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o287 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o293 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o255 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o256 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o257 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o258 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o259 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o260 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o261 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o262 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o263 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Command to send: m d o264 e (clientserver.py:501, send_command) 2026-04-02 10:29:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o265 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o266 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o267 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o268 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o269 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o270 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o271 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o272 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o273 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o275 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o276 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o277 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o278 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o280 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o281 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o282 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o283 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o284 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o286 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o288 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o289 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o291 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o292 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o294 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o295 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o296 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o300 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Command to send: m d o305 e (clientserver.py:501, send_command) 2026-04-02 10:29:50 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) ________________________ test_startup_without_zookeeper ________________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_startup_without_zookeeper(start_cluster): node1.query( "INSERT INTO test_table VALUES ('2018-10-01', 1), ('2018-10-02', 2), ('2018-10-03', 3)" ) assert node1.query("SELECT COUNT(*) from test_table") == "3\n" assert ( node1.query("SELECT is_readonly from system.replicas where table='test_table'") == "0\n" ) > cluster.run_kazoo_commands_with_retries(drop_zk) test_replication_without_zookeeper/test.py:51: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3317: in run_kazoo_commands_with_retries kazoo_callback(zk) test_replication_without_zookeeper/test.py:38: in drop_zk zk.delete(path="/clickhouse", recursive=True) /usr/local/lib/python3.10/dist-packages/kazoo/client.py:1431: in delete return self._delete_recursive(path) /usr/local/lib/python3.10/dist-packages/kazoo/client.py:1466: in _delete_recursive self.delete(path) /usr/local/lib/python3.10/dist-packages/kazoo/client.py:1433: in delete return self.delete_async(path, version).get() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = block = True, timeout = None def get(self, block=True, timeout=None): """Return the stored value or raise the exception. If there is no value raises TimeoutError. """ with self._condition: if self._exception is not _NONE: if self._exception is None: return self.value raise self._exception elif block: self._condition.wait(timeout) if self._exception is not _NONE: if self._exception is None: return self.value > raise self._exception E kazoo.exceptions.NotEmptyError /usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:86: NotEmptyError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-04-02 10:35:55 [ 415 ] INFO : Running tests in /ClickHouse/tests/integration/test_replication_without_zookeeper/test.py (cluster.py:2788, start) 2026-04-02 10:35:55 [ 415 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start) 2026-04-02 10:35:55 [ 415 ] DEBUG : Docker networks for project roottestreplicationwithoutzookeeper_gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:35:55 [ 415 ] DEBUG : Docker containers for project roottestreplicationwithoutzookeeper_gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:35:55 [ 415 ] DEBUG : Docker volumes for project roottestreplicationwithoutzookeeper_gw9 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:35:55 [ 415 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:35:55 [ 415 ] DEBUG : Docker networks for project roottestreplicationwithoutzookeeper_gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:35:56 [ 415 ] DEBUG : Docker containers for project roottestreplicationwithoutzookeeper_gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:35:56 [ 415 ] DEBUG : Docker volumes for project roottestreplicationwithoutzookeeper_gw9 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:35:56 [ 415 ] DEBUG : Command:docker container list --all --filter name='^/roottestreplicationwithoutzookeeper_gw9_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:35:56 [ 415 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:35:56 [ 415 ] DEBUG : No running containers for project: roottestreplicationwithoutzookeeper_gw9 (cluster.py:904, cleanup) 2026-04-02 10:35:56 [ 415 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:35:56 [ 415 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:35:56 [ 415 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:35:57 [ 415 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:35:57 [ 415 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:35:57 [ 415 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:35:57 [ 415 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:35:57 [ 415 ] DEBUG : Stdout:9 (cluster.py:121, run_and_check) 2026-04-02 10:35:57 [ 415 ] DEBUG : Setup directory for instance: node1 (cluster.py:2808, start) 2026-04-02 10:35:57 [ 415 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_replication_without_zookeeper/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/configs/config.d (cluster.py:4632, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/database (cluster.py:4649, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/logs (cluster.py:4660, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon; coproc tail -f /dev/null; wait $$!" (cluster.py:4746, create_dir) 2026-04-02 10:35:57 [ 415 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:1e0b53d756cf', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env (cluster.py:86, _create_env_file) 2026-04-02 10:35:57 [ 415 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:35:57 [ 415 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:35:57 [ 415 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-02 10:35:57 [ 415 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-02 10:35:57 [ 415 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request) 2026-04-02 10:35:57 [ 415 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', '--project-name', 'roottestreplicationwithoutzookeeper_gw9', '--file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', 'pull'] (cluster.py:113, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo2 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo3 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling node1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling node1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo3 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling node1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling node1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling node1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo3 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo2 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo2 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Stderr:Pulling zoo1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:12 [ 415 ] DEBUG : Setup ZooKeeper (cluster.py:2849, start) 2026-04-02 10:36:12 [ 415 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/coordination'] (cluster.py:2850, start) 2026-04-02 10:36:12 [ 415 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', '--project-name', 'roottestreplicationwithoutzookeeper_gw9', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--verbose', 'up', '-d'] (cluster.py:113, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.config.config.find: Using configuration files: /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.docker_client.get_client: docker-compose version 1.29.2, build unknown (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:docker-py version: (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:CPython version: 3.10.12 (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:OpenSSL version: OpenSSL 3.0.2 15 Mar 2022 (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '23.0.6', 'Details': {'ApiVersion': '1.42', 'Arch': 'amd64', 'BuildTime': '2023-05-05T21:18:13.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9dbdbd4', 'GoVersion': 'go1.19.9', 'KernelVersion': '5.15.0-130-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.7.25', 'Details': {'GitCommit': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}}, {'Name': 'runc', 'Version': '1.2.4', 'Details': {'GitCommit': 'v1.2.4-0-g6c52b3f'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=23.0.6, ApiVersion=1.42, MinAPIVersion=1.12, GitCommit=9dbdbd4, GoVersion=go1.19.9, Os=linux, Arch=amd64, KernelVersion=5.15.0-130-generic, BuildTime=2023-05-05T21:18:13.000000000+00:00 (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestreplicationwithoutzookeepergw9_default') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info <- () (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker info -> {'Architecture': 'x86_64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'BridgeNfIp6tables': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'BridgeNfIptables': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CPUSet': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CPUShares': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CgroupDriver': 'cgroupfs', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CgroupVersion': '2', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'ContainerdCommit': {'Expected': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'ID': 'bcc810d6b9066471b0b6fa75f557a15a1cbf31bb'}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Containers': 43, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('roottestreplicationwithoutzookeeper_gw9_default') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.network.ensure: Creating network "roottestreplicationwithoutzookeeper_gw9_default" with the default driver (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network <- (name='roottestreplicationwithoutzookeeper_gw9_default', driver=None, options=None, ipam=None, internal=False, enable_ipv6=False, labels={'com.docker.compose.project': 'roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.network': 'default', 'com.docker.compose.version': '1.29.2'}, attachable=True, check_duplicate=True) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_network -> {'Id': 'd9f6bbe385812bb84e69f44f3f0332ed113a193755252abcaf05cbdbcc0b3189', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Warning': ''} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {, , } (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo1', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo2', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo2', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=roottestreplicationwithoutzookeepergw9', 'com.docker.compose.service=zoo3', 'com.docker.compose.oneoff=False']}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo1', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 3607e41ee3a5d718029e9c39dc73ff93791fa2c6e1bafc4b270c1ba6b4dc67c2 (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestreplicationwithoutzookeeper_gw9_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: {ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo3', number=1)} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Starting producer thread for ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo3', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Source': '/clickhouse', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Target': '/usr/bin/clickhouse-keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 60cd8e797ea55fe7aa6f412a20306ea0f542fb62a079acc60669bd18508bd7f0 (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config2.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestreplicationwithoutzookeeper_gw9_zoo2_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service': 'zoo2', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '3607e41ee3a5d718029e9c39dc73ff93791fa2c6e1bafc4b270c1ba6b4dc67c2'}, host_config={'NetworkMode': 'roottestreplicationwithoutzookeeper_gw9_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper2/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestreplicationwithoutzookeeper_gw9_default': {'Aliases': ['zoo2'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestreplicationwithoutzookeeper_gw9_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Source': '/clickhouse', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Target': '/usr/bin/clickhouse-keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config1.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestreplicationwithoutzookeeper_gw9_zoo1_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service': 'zoo1', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '60cd8e797ea55fe7aa6f412a20306ea0f542fb62a079acc60669bd18508bd7f0'}, host_config={'NetworkMode': 'roottestreplicationwithoutzookeeper_gw9_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestreplicationwithoutzookeeper_gw9_default': {'Aliases': ['zoo1'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('altinityinfra/integration-test:1e0b53d756cf') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Author': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Comment': 'buildkit.dockerfile.v0', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'ArgsEscaped': True, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': ['/bin/sh', '-c', 'sleep 1'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Entrypoint': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.service.build_container_labels: Added config hash: 63d89da0ef5173b0b96b03ca812a1c5c6b60c13183a747163b7d98abe91989ea (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='roottestreplicationwithoutzookeeper_gw9_default', devices=None, device_requests=None, dns=None, dns_opt=['attempts:2', 'timeout:1', 'inet6', 'rotate'], dns_search=None, restart_policy={'Name': 'always', 'MaximumRetryCount': 0}, runtime=None, cap_add=['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=['label:disable'], ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=[{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}], device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Links': [], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'LogConfig': {'Config': {}, 'Type': ''}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Mounts': [{'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/log', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Target': '/var/log/clickhouse-keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Type': 'bind'}, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: {'ReadOnly': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container <- (entrypoint='clickhouse keeper --config=/etc/clickhouse-keeper/keeper_config3.xml --log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log --errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log', image='altinityinfra/integration-test:1e0b53d756cf', user='0', volumes={}, name='roottestreplicationwithoutzookeeper_gw9_zoo3_1', detach=True, environment=[], labels={'com.docker.compose.project': 'roottestreplicationwithoutzookeeper_gw9', 'com.docker.compose.service': 'zoo3', 'com.docker.compose.oneoff': 'False', 'com.docker.compose.project.working_dir': '/ClickHouse/tests/integration/compose', 'com.docker.compose.project.config_files': '/ClickHouse/tests/integration/compose/docker_compose_keeper.yml', 'com.docker.compose.project.environment_file': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', 'com.docker.compose.container-number': '1', 'com.docker.compose.version': '1.29.2', 'com.docker.compose.config-hash': '63d89da0ef5173b0b96b03ca812a1c5c6b60c13183a747163b7d98abe91989ea'}, host_config={'NetworkMode': 'roottestreplicationwithoutzookeeper_gw9_default', 'RestartPolicy': {'Name': 'always', 'MaximumRetryCount': 0}, 'CapAdd': ['SYS_PTRACE', 'NET_ADMIN', 'IPC_LOCK', 'SYS_NICE'], 'DnsOptions': ['attempts:2', 'timeout:1', 'inet6', 'rotate'], 'SecurityOpt': ['label:disable'], 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}, 'Mounts': [{'Target': '/var/log/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/log', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper1/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/etc/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/config', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/var/lib/clickhouse-keeper', 'Source': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/keeper3/coordination', 'Type': 'bind', 'ReadOnly': None}, {'Target': '/usr/bin/clickhouse-keeper', 'Source': '/clickhouse', 'Type': 'bind', 'ReadOnly': None}]}, networking_config={'EndpointsConfig': {'roottestreplicationwithoutzookeeper_gw9_default': {'Aliases': ['zoo3'], 'IPAMConfig': {}}}}) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'bdbb1bccba122d9411acd41a8719b75f4f3c3ddb30cc181166f5cee0aa6e38fd', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('bdbb1bccba122d9411acd41a8719b75f4f3c3ddb30cc181166f5cee0aa6e38fd') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'd82e14790168644d740d6aa3a58a4023ce7c94b7f8146ef30ac29cb58d2f5595', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('d82e14790168644d740d6aa3a58a4023ce7c94b7f8146ef30ac29cb58d2f5595') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config1.xml', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('bdbb1bccba122d9411acd41a8719b75f4f3c3ddb30cc181166f5cee0aa6e38fd', 'roottestreplicationwithoutzookeeper_gw9_default') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': '10bedbbf3b5f50969161ace6636c2e92ef252a2485052a69fceca53b9b1f82a2', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Warnings': []} (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('10bedbbf3b5f50969161ace6636c2e92ef252a2485052a69fceca53b9b1f82a2') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config3.xml', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('d82e14790168644d740d6aa3a58a4023ce7c94b7f8146ef30ac29cb58d2f5595', 'roottestreplicationwithoutzookeeper_gw9_default') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('bdbb1bccba122d9411acd41a8719b75f4f3c3ddb30cc181166f5cee0aa6e38fd', 'roottestreplicationwithoutzookeeper_gw9_default', aliases=['zoo1', 'bdbb1bccba12'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Args': ['keeper', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--config=/etc/clickhouse-keeper/keeper_config2.xml', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--log-file=/var/log/clickhouse-keeper/clickhouse-keeper.log', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: '--errorlog-file=/var/log/clickhouse-keeper/clickhouse-keeper.err.log'], (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Config': {'AttachStderr': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdin': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'AttachStdout': False, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Cmd': None, (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr: 'Domainname': '', (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:... (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('10bedbbf3b5f50969161ace6636c2e92ef252a2485052a69fceca53b9b1f82a2', 'roottestreplicationwithoutzookeeper_gw9_default') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('d82e14790168644d740d6aa3a58a4023ce7c94b7f8146ef30ac29cb58d2f5595', 'roottestreplicationwithoutzookeeper_gw9_default', aliases=['zoo3', 'd82e14790168'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('bdbb1bccba122d9411acd41a8719b75f4f3c3ddb30cc181166f5cee0aa6e38fd') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('10bedbbf3b5f50969161ace6636c2e92ef252a2485052a69fceca53b9b1f82a2', 'roottestreplicationwithoutzookeeper_gw9_default', aliases=['10bedbbf3b5f', 'zoo2'], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('10bedbbf3b5f50969161ace6636c2e92ef252a2485052a69fceca53b9b1f82a2') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start <- ('d82e14790168644d740d6aa3a58a4023ce7c94b7f8146ef30ac29cb58d2f5595') (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo1', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo3', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.cli.verbose_proxy.proxy_callable: docker start -> None (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='roottestreplicationwithoutzookeeper_gw9', service='zoo2', number=1) (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.parallel_execute_iter: Finished processing: (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Stderr:compose.parallel.feed_queue: Pending: set() (cluster.py:123, run_and_check) 2026-04-02 10:36:27 [ 415 ] DEBUG : Wait ZooKeeper to start (cluster.py:2504, wait_zookeeper_to_start) 2026-04-02 10:36:27 [ 415 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:36:27 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_zoo1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:27 [ 415 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.21.2, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:36:27 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:27 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:36:27 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:27 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:36:27 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:27 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:36:28 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:28 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:36:29 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:29 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:36:32 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:32 [ 415 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:36:32 [ 415 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:36:32 [ 415 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:32 [ 415 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:36:32 [ 415 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:36:32 [ 415 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:36:32 [ 415 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:36:32 [ 415 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:36:32 [ 415 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:36:32 [ 415 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:36:32 [ 415 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2135, get_instance_ip) 2026-04-02 10:36:32 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_zoo2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:32 [ 415 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.21.3, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:36:32 [ 415 ] INFO : Connecting to 172.16.21.3(172.16.21.3):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:32 [ 415 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:36:33 [ 415 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:36:33 [ 415 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:36:33 [ 415 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:36:33 [ 415 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:36:33 [ 415 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:36:33 [ 415 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:36:33 [ 415 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2135, get_instance_ip) 2026-04-02 10:36:33 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_zoo3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:33 [ 415 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.21.4, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:36:33 [ 415 ] INFO : Connecting to 172.16.21.4(172.16.21.4):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:33 [ 415 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:36:33 [ 415 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-04-02 10:36:33 [ 415 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-04-02 10:36:33 [ 415 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-04-02 10:36:33 [ 415 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:36:33 [ 415 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:36:34 [ 415 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-04-02 10:36:34 [ 415 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-04-02 10:36:34 [ 415 ] DEBUG : All instances of ZooKeeper Secure started (cluster.py:2519, wait_zookeeper_nodes_to_start) 2026-04-02 10:36:34 [ 415 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env --project-name roottestreplicationwithoutzookeeper_gw9 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') (cluster.py:3146, start) 2026-04-02 10:36:34 [ 415 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', '--project-name', 'roottestreplicationwithoutzookeeper_gw9', '--file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-04-02 10:36:42 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:36:42 [ 415 ] DEBUG : Stderr:Creating roottestreplicationwithoutzookeeper_gw9_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:36:42 [ 415 ] DEBUG : ClickHouse instance created (cluster.py:3154, start) 2026-04-02 10:36:42 [ 415 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.21.5... (cluster.py:3161, start) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:42 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:43 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:44 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:45 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:46 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:47 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:48 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:49 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:50 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:51 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:52 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/3bf2f88994fba98bbb7477504bb649b68c8f526dcea4764d5ccb2e9e5150c584/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:53 [ 415 ] DEBUG : ClickHouse node1 started (cluster.py:3165, start) 2026-04-02 10:36:53 [ 415 ] DEBUG : Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated', 'node1') ORDER BY id PARTITION BY toYYYYMM(date); on node1 (cluster.py:3602, query) ------------------------------ Captured log call ------------------------------- 2026-04-02 10:36:54 [ 415 ] DEBUG : Executing query INSERT INTO test_table VALUES ('2018-10-01', 1), ('2018-10-02', 2), ('2018-10-03', 3) on node1 (cluster.py:3602, query) 2026-04-02 10:36:55 [ 415 ] DEBUG : Executing query SELECT COUNT(*) from test_table on node1 (cluster.py:3602, query) 2026-04-02 10:36:57 [ 415 ] DEBUG : Executing query SELECT is_readonly from system.replicas where table='test_table' on node1 (cluster.py:3602, query) 2026-04-02 10:36:58 [ 415 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2135, get_instance_ip) 2026-04-02 10:36:58 [ 415 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestreplicationwithoutzookeeper_gw9_zoo1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-02 10:36:58 [ 415 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.21.2, port:2181, use_ssl:False (cluster.py:3286, get_kazoo_client) 2026-04-02 10:36:58 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-04-02 10:36:58 [ 415 ] DEBUG : run_kazoo_commands_with_retries: zoo1, (cluster.py:3307, run_kazoo_commands_with_retries) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=1): GetChildren(path='/clickhouse', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=1): ['task_queue', 'sessions', 'tables'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=2): GetChildren(path='/clickhouse/task_queue', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=2): ['ddl'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=3): GetChildren(path='/clickhouse/task_queue/ddl', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=3): [] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=4): Delete(path='/clickhouse/task_queue/ddl', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=4): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=5): Delete(path='/clickhouse/task_queue', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=5): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=6): GetChildren(path='/clickhouse/sessions', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=6): ['zookeeper'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=7): GetChildren(path='/clickhouse/sessions/zookeeper', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=7): ['77177845-85f1-4ab9-a30f-f3fe829d4323'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=8): GetChildren(path='/clickhouse/sessions/zookeeper/77177845-85f1-4ab9-a30f-f3fe829d4323', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=8): [] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=9): Delete(path='/clickhouse/sessions/zookeeper/77177845-85f1-4ab9-a30f-f3fe829d4323', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=9): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=10): Delete(path='/clickhouse/sessions/zookeeper', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=10): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=11): Delete(path='/clickhouse/sessions', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=11): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=12): GetChildren(path='/clickhouse/tables', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=12): ['test'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=13): GetChildren(path='/clickhouse/tables/test', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=13): ['replicated'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=14): GetChildren(path='/clickhouse/tables/test/replicated', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=14): ['quorum', 'async_blocks', 'lost_part_count', 'mutations', 'pinned_part_uuids', 'block_numbers', 'metadata', 'temp', 'table_shared_id', 'alter_partition_version', 'replicas', 'leader_election', 'nonincrement_block_numbers', 'columns', 'blocks', 'log', 'part_moves_shard'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=15): GetChildren(path='/clickhouse/tables/test/replicated/quorum', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=15): ['parallel', 'last_part', 'failed_parts'] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=16): GetChildren(path='/clickhouse/tables/test/replicated/quorum/parallel', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=16): [] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=17): Delete(path='/clickhouse/tables/test/replicated/quorum/parallel', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=17): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=18): GetChildren(path='/clickhouse/tables/test/replicated/quorum/last_part', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=18): [] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=19): Delete(path='/clickhouse/tables/test/replicated/quorum/last_part', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=19): True (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=20): GetChildren(path='/clickhouse/tables/test/replicated/quorum/failed_parts', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:58 [ 415 ] DEBUG : Received response(xid=20): [] (connection.py:410, _read_response) 2026-04-02 10:36:58 [ 415 ] DEBUG : Sending request(xid=21): Delete(path='/clickhouse/tables/test/replicated/quorum/failed_parts', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=21): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=22): Delete(path='/clickhouse/tables/test/replicated/quorum', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=22): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=23): GetChildren(path='/clickhouse/tables/test/replicated/async_blocks', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=23): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=24): Delete(path='/clickhouse/tables/test/replicated/async_blocks', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=24): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=25): GetChildren(path='/clickhouse/tables/test/replicated/lost_part_count', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=25): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=26): Delete(path='/clickhouse/tables/test/replicated/lost_part_count', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=26): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=27): GetChildren(path='/clickhouse/tables/test/replicated/mutations', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=27): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=28): Delete(path='/clickhouse/tables/test/replicated/mutations', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=28): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=29): GetChildren(path='/clickhouse/tables/test/replicated/pinned_part_uuids', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=29): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=30): Delete(path='/clickhouse/tables/test/replicated/pinned_part_uuids', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=30): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=31): GetChildren(path='/clickhouse/tables/test/replicated/block_numbers', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=31): ['201810'] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=32): GetChildren(path='/clickhouse/tables/test/replicated/block_numbers/201810', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=32): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=33): Delete(path='/clickhouse/tables/test/replicated/block_numbers/201810', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=33): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=34): Delete(path='/clickhouse/tables/test/replicated/block_numbers', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=34): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=35): GetChildren(path='/clickhouse/tables/test/replicated/metadata', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=35): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=36): Delete(path='/clickhouse/tables/test/replicated/metadata', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=36): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=37): GetChildren(path='/clickhouse/tables/test/replicated/temp', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=37): ['abandonable_lock-other', 'abandonable_lock-insert'] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=38): GetChildren(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-other', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=38): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=39): Delete(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-other', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=39): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=40): GetChildren(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-insert', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=40): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=41): Delete(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-insert', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=41): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=42): Delete(path='/clickhouse/tables/test/replicated/temp', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=42): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=43): GetChildren(path='/clickhouse/tables/test/replicated/table_shared_id', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=43): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=44): Delete(path='/clickhouse/tables/test/replicated/table_shared_id', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=44): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=45): GetChildren(path='/clickhouse/tables/test/replicated/alter_partition_version', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=45): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=46): Delete(path='/clickhouse/tables/test/replicated/alter_partition_version', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=46): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=47): GetChildren(path='/clickhouse/tables/test/replicated/replicas', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=47): ['node1'] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=48): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=48): ['metadata', 'queue', 'metadata_version', 'is_active', 'host', 'min_unprocessed_insert_time', 'is_lost', 'mutation_pointer', 'parts', 'columns', 'flags', 'max_processed_insert_time', 'log_pointer', 'creator_info'] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=49): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/metadata', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=49): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=50): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/metadata', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=50): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=51): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/queue', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=51): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=52): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/queue', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=52): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=53): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/metadata_version', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=53): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=54): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/metadata_version', version=-1) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=54): True (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=55): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/is_active', watcher=None) (connection.py:312, _submit) 2026-04-02 10:36:59 [ 415 ] DEBUG : Received response(xid=55): [] (connection.py:410, _read_response) 2026-04-02 10:36:59 [ 415 ] DEBUG : Sending request(xid=56): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/is_active', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=56): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=57): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/host', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=57): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=58): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/host', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=58): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=59): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/min_unprocessed_insert_time', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=59): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=60): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/min_unprocessed_insert_time', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=60): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=61): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/is_lost', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=61): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=62): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/is_lost', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=62): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=63): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/mutation_pointer', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=63): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=64): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/mutation_pointer', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=64): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=65): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/parts', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=65): ['201810_0_0_0'] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=66): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/parts/201810_0_0_0', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=66): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=67): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/parts/201810_0_0_0', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=67): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=68): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/parts', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=68): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=69): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/columns', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=69): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=70): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/columns', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=70): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=71): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/flags', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=71): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=72): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/flags', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=72): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=73): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/max_processed_insert_time', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=73): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=74): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/max_processed_insert_time', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=74): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=75): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/log_pointer', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=75): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=76): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/log_pointer', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=76): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=77): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/creator_info', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=77): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=78): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/creator_info', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=78): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=79): Delete(path='/clickhouse/tables/test/replicated/replicas/node1', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=79): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=80): Delete(path='/clickhouse/tables/test/replicated/replicas', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=80): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=81): GetChildren(path='/clickhouse/tables/test/replicated/leader_election', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=81): ['leader_election-0'] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=82): GetChildren(path='/clickhouse/tables/test/replicated/leader_election/leader_election-0', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=82): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=83): Delete(path='/clickhouse/tables/test/replicated/leader_election/leader_election-0', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=83): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=84): Delete(path='/clickhouse/tables/test/replicated/leader_election', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=84): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=85): GetChildren(path='/clickhouse/tables/test/replicated/nonincrement_block_numbers', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=85): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=86): Delete(path='/clickhouse/tables/test/replicated/nonincrement_block_numbers', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=86): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=87): GetChildren(path='/clickhouse/tables/test/replicated/columns', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=87): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=88): Delete(path='/clickhouse/tables/test/replicated/columns', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=88): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=89): GetChildren(path='/clickhouse/tables/test/replicated/blocks', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=89): ['201810_2956868034535131113_3759223844523231509'] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=90): GetChildren(path='/clickhouse/tables/test/replicated/blocks/201810_2956868034535131113_3759223844523231509', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=90): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=91): Delete(path='/clickhouse/tables/test/replicated/blocks/201810_2956868034535131113_3759223844523231509', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=91): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=92): Delete(path='/clickhouse/tables/test/replicated/blocks', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=92): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=93): GetChildren(path='/clickhouse/tables/test/replicated/log', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=93): ['log-0000000000'] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=94): GetChildren(path='/clickhouse/tables/test/replicated/log/log-0000000000', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=94): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=95): Delete(path='/clickhouse/tables/test/replicated/log/log-0000000000', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=95): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=96): Delete(path='/clickhouse/tables/test/replicated/log', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=96): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=97): GetChildren(path='/clickhouse/tables/test/replicated/part_moves_shard', watcher=None) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=97): [] (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=98): Delete(path='/clickhouse/tables/test/replicated/part_moves_shard', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=98): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=99): Delete(path='/clickhouse/tables/test/replicated', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=99): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=100): Delete(path='/clickhouse/tables/test', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=100): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=101): Delete(path='/clickhouse/tables', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:00 [ 415 ] DEBUG : Received response(xid=101): True (connection.py:410, _read_response) 2026-04-02 10:37:00 [ 415 ] DEBUG : Sending request(xid=102): Delete(path='/clickhouse', version=-1) (connection.py:312, _submit) 2026-04-02 10:37:01 [ 415 ] DEBUG : Received error(xid=102) NotEmptyError() (connection.py:392, _read_response) ---------------------------- Captured log teardown ----------------------------- 2026-04-02 10:37:06 [ 415 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', '--project-name', 'roottestreplicationwithoutzookeeper_gw9', '--file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-04-02 10:37:16 [ 415 ] WARNING : Connection dropped: outstanding heartbeat ping not received (connection.py:622, _connect_attempt) 2026-04-02 10:37:16 [ 415 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-04-02 10:37:16 [ 415 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-04-02 10:37:16 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:37:16 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:37:16 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:37:16 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:37:17 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:37:17 [ 415 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-04-02 10:37:17 [ 415 ] INFO : Connecting to 172.16.21.2(172.16.21.2):2181, use_ssl: False (connection.py:650, _connect) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Stderr:Stopping roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:18 [ 415 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-04-02 10:37:19 [ 415 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/.env', '--project-name', 'roottestreplicationwithoutzookeeper_gw9', '--file', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances_0_gw9/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing roottestreplicationwithoutzookeeper_gw9_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Stderr:Removing network roottestreplicationwithoutzookeeper_gw9_default (cluster.py:123, run_and_check) 2026-04-02 10:37:23 [ 415 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:37:24 [ 415 ] DEBUG : Docker networks for project roottestreplicationwithoutzookeeper_gw9 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:37:24 [ 415 ] DEBUG : Docker containers for project roottestreplicationwithoutzookeeper_gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:37:24 [ 415 ] DEBUG : Docker volumes for project roottestreplicationwithoutzookeeper_gw9 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:37:24 [ 415 ] DEBUG : Command:docker container list --all --filter name='^/roottestreplicationwithoutzookeeper_gw9_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:37:25 [ 415 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : No running containers for project: roottestreplicationwithoutzookeeper_gw9 (cluster.py:904, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:37:25 [ 415 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:37:25 [ 415 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:37:25 [ 415 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:37:25 [ 415 ] DEBUG : Stdout:7 (cluster.py:121, run_and_check) ______________________________ test_remove_policy ______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remove_policy(started_cluster): try: name = "test_remove_policy" engine = "MergeTree()" start_over() add_disk(node1, "jbod3", "/jbod3/") add_disk(node1, "jbod4", "/jbod4/") add_policy(node1, "remove_policy_cool_policy", {"volume1": ["jbod3", "jbod4"]}) > node1.restart_clickhouse(kill=True) test_reloading_storage_configuration/test.py:559: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:32:24 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:24 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:32:25 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:32:25 [ 328 ] DEBUG : Stdout: 6098 ? 00:00:41 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:32:25 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:25 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'pkill -9 clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:32:27 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:27 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:28 [ 328 ] DEBUG : Stdout:6098 (cluster.py:121, run_and_check) 2026-04-02 10:32:29 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:29 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:30 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:30 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:32 [ 328 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3964, start_clickhouse) 2026-04-02 10:32:32 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:32 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2026-04-02 10:32:36 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:36 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:37 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:32:37 [ 328 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:32:37 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:32:37 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:32:42 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:32:42 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:46 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:49 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:52 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:53 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:57 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:32:58 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:01 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:05 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:08 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:12 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:12 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:14 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:33:14 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:33:14 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:14 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:16 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:33:16 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:19 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:21 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:23 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:28 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:31 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:35 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:37 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:45 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:49 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:33:51 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:51 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:54 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:33:54 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:33:54 [ 328 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:33:54 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:54 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:33:56 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:33:56 [ 328 ] DEBUG : Stdout: 6915 ? 00:00:33 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:33:56 [ 328 ] INFO : PS RESULT: PID TTY TIME CMD 6915 ? 00:00:33 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:33:56 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:56 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:33:57 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:33:57 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:2173, exec_in_container) 2026-04-02 10:33:57 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:113, run_and_check) 2026-04-02 10:38:57 [ 328 ] WARNING : Current start attempt failed. Will kill 6915 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:38:57 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 6915'] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:57 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'kill -9 6915'] (cluster.py:113, run_and_check) 2026-04-02 10:38:59 [ 328 ] DEBUG : Executing query DROP TABLE IF EXISTS test_remove_policy on node1 (cluster.py:3602, query) ___________________ test_table_invalid_column_family_options ___________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_table_invalid_column_family_options(start_cluster): node.exec_in_container( [ "bash", "-c", "sed -i 's/max_bytes_for_level_base/no_such_table_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml", ] ) > node.restart_clickhouse() test_rocksdb_options/test.py:194: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:36:52 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/max_bytes_for_level_base/no_such_table_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:52 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "sed -i 's/max_bytes_for_level_base/no_such_table_column_family_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:113, run_and_check) 2026-04-02 10:36:53 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:53 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:36:54 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:36:54 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:36:54 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:54 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:36:54 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:54 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:55 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:56 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:56 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:57 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:36:58 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:36:58 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:36:59 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:00 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:00 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:00 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:01 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:01 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:02 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:03 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:03 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:04 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:05 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:05 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:06 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:07 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:07 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:08 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:09 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:09 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:10 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:11 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:11 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:11 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:12 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:12 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:13 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:15 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:16 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:16 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:17 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:18 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:18 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:19 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:19 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:21 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:22 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:22 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:23 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:24 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:24 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:25 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:26 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:26 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:26 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:27 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:27 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:28 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:29 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:29 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:30 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:31 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:31 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:32 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:33 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:33 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:34 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:35 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:35 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:35 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:36 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:37 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:38 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:39 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:39 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:40 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:41 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:41 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:43 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:45 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:46 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:46 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:47 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:48 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:48 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:49 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:50 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:50 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:50 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:51 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:51 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:52 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:53 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:53 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:54 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:55 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:55 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:56 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:56 [ 334 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:2385 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:37:56 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:56 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:37:57 [ 334 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:37:57 [ 334 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:37:57 [ 334 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:37:57 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:57 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:57 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:57 [ 334 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:37:57 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:37:57 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:37:58 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:37:58 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:37:59 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:01 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:02 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:04 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:05 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:06 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:07 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:09 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:11 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:12 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:12 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:13 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:13 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:38:13 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:13 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:13 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:13 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:14 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:16 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:17 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:19 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:21 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:23 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:25 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:27 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:28 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:29 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:29 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:30 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:30 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:38:30 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:30 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:31 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:31 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:32 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:33 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:35 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:36 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:37 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:38 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:39 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:40 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:42 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:43 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:43 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:44 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:44 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:38:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:38:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:38:45 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:38:45 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:46 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:50 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:51 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:53 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:55 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:57 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:38:59 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:39:01 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:39:02 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:39:04 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:04 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:06 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:06 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:39:06 [ 334 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:39:06 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:06 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:39:08 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:39:08 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:39:08 [ 334 ] INFO : PS RESULT: PID TTY TIME CMD 2385 ? 00:00:35 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:39:08 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:08 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:09 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:09 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:09 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:113, run_and_check) 2026-04-02 10:39:11 [ 334 ] DEBUG : Stderr:warning: process 2385 is already traced by process 3128process 2385 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:39:11 [ 334 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:39:11 [ 334 ] WARNING : Current start attempt failed. Will kill 2385 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:39:11 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2385'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:11 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'kill -9 2385'] (cluster.py:113, run_and_check) ________________________ test_remove_volume_from_policy ________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remove_volume_from_policy(started_cluster): try: name = "test_remove_volume_from_policy" engine = "MergeTree()" start_over() add_disk(node1, "jbod3", "/jbod3/") add_disk(node1, "jbod4", "/jbod4/") add_policy( node1, "test_remove_volume_from_policy_cool_policy", collections.OrderedDict([("volume1", ["jbod3"]), ("volume2", ["jbod4"])]), ) > node1.restart_clickhouse(kill=True) test_reloading_storage_configuration/test.py:608: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:39:03 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:03 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:39:04 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:39:04 [ 328 ] DEBUG : Stdout: 6915 ? 00:00:37 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:39:04 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:04 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'pkill -9 clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:39:06 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:06 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:08 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:09 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:09 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:11 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:12 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:12 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:13 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:14 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:14 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:16 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:17 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:17 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:20 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:21 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:21 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:22 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:23 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:23 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:25 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:26 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:26 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:28 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:29 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:29 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:30 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:31 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:31 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:33 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:34 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:34 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:36 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:37 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:37 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:37 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:38 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:39 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:40 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:40 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:43 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:44 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:44 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:45 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:46 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:46 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:47 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:48 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:48 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:49 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:50 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:50 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:52 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:53 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:53 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:54 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:55 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:55 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:56 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:57 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:57 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:57 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:39:58 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:58 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:01 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:02 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:02 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:03 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:04 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:04 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:05 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:06 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:06 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:07 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:08 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:08 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:09 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:09 [ 328 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:6915 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:40:09 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:09 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:40:09 [ 328 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:40:09 [ 328 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:40:09 [ 328 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:40:09 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:09 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:10 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:10 [ 328 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:40:10 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:10 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:11 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:11 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:13 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:14 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:15 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:17 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:18 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:20 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:21 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:23 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:25 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:28 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:28 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:28 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:28 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:40:28 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:28 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:30 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:30 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:31 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:35 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:37 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:38 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:39 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:41 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:43 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:44 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:46 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:48 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:48 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:48 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:48 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:40:48 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:48 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:49 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:40:49 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:50 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:51 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:53 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:54 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:56 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:57 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:40:59 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:00 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:02 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:05 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:05 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:06 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:06 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:41:06 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:06 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:06 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:06 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:08 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:09 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:11 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:12 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:13 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:14 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:16 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:17 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:18 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:41:19 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:19 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:20 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:20 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:41:20 [ 328 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:41:20 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:20 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:41:21 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:41:21 [ 328 ] DEBUG : Stdout: 6915 ? 00:00:37 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:41:21 [ 328 ] INFO : PS RESULT: PID TTY TIME CMD 6915 ? 00:00:37 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:41:21 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:21 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:21 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:21 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:21 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:113, run_and_check) 2026-04-02 10:41:22 [ 328 ] DEBUG : Stderr:warning: process 6915 is already traced by process 7673process 6915 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:41:22 [ 328 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:41:22 [ 328 ] WARNING : Current start attempt failed. Will kill 6915 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:41:22 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 6915'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:22 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'kill -9 6915'] (cluster.py:113, run_and_check) 2026-04-02 10:41:23 [ 328 ] DEBUG : Executing query DROP TABLE IF EXISTS test_remove_volume_from_policy on node1 (cluster.py:3602, query) __________________________ test_table_invalid_options __________________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_table_invalid_options(start_cluster): node.exec_in_container( [ "bash", "-c", "sed -i 's/max_open_files/no_such_table_option/g' /etc/clickhouse-server/config.d/rocksdb.xml", ] ) > node.restart_clickhouse() test_rocksdb_options/test.py:101: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:39:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/max_open_files/no_such_table_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "sed -i 's/max_open_files/no_such_table_option/g' /etc/clickhouse-server/config.d/rocksdb.xml"] (cluster.py:113, run_and_check) 2026-04-02 10:39:16 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:16 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:39:18 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:39:18 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:39:18 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:18 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:39:19 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:19 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:20 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:21 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:21 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:23 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:24 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:24 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:26 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:27 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:27 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:28 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:29 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:29 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:32 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:33 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:33 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:35 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:36 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:36 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:37 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:38 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:38 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:39 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:40 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:40 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:41 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:42 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:42 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:43 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:44 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:44 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:45 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:46 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:46 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:48 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:49 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:49 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:50 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:51 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:51 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:52 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:53 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:53 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:54 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:55 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:55 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:56 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:57 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:57 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:39:58 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:39:59 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:39:59 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:01 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:02 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:02 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:03 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:04 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:04 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:05 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:06 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:06 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:07 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:08 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:08 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:09 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:10 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:10 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:11 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:12 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:12 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:13 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:14 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:15 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:15 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:16 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:17 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:17 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:18 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:19 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:19 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:20 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:20 [ 334 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:2385 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:40:20 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:20 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:40:20 [ 334 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:40:20 [ 334 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:40:20 [ 334 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385 > /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:40:20 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:20 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:21 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:21 [ 334 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:40:21 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:21 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:22 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:22 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:23 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:25 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:28 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:29 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:31 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:32 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:37 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:39 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:40 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:41 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:41 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:42 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:42 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:40:42 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:42 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:43 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:43 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:45 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:46 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:47 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:48 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:49 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:51 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:52 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:54 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:56 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:40:57 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:57 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:58 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:58 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:40:58 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:40:58 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:40:59 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:40:59 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:00 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:02 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:04 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:06 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:07 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:09 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:11 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:12 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:13 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:14 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:41:14 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:41:14 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:14 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:15 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:41:15 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:16 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:18 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:20 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:21 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:23 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:24 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:26 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:27 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:28 [ 334 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-02 10:41:30 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:30 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:30 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:41:30 [ 334 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:41:30 [ 334 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:41:30 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:30 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:41:31 [ 334 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:41:31 [ 334 ] DEBUG : Stdout: 2385 ? 00:00:35 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:41:31 [ 334 ] INFO : PS RESULT: PID TTY TIME CMD 2385 ? 00:00:35 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:41:31 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:31 [ 334 ] DEBUG : Command:['docker', 'exec', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:31 [ 334 ] DEBUG : Stdout:2385 (cluster.py:121, run_and_check) 2026-04-02 10:41:31 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:31 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 2385"] (cluster.py:113, run_and_check) 2026-04-02 10:41:32 [ 334 ] DEBUG : Stderr:warning: process 2385 is already traced by process 3128process 2385 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:41:32 [ 334 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:41:32 [ 334 ] WARNING : Current start attempt failed. Will kill 2385 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:41:32 [ 334 ] DEBUG : run container_id:roottestrocksdboptions_gw2_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 2385'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:32 [ 334 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestrocksdboptions_gw2_node_1', 'bash', '-c', 'kill -9 2385'] (cluster.py:113, run_and_check) ____________________ test_table_valid_column_family_options ____________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_table_valid_column_family_options(start_cluster): > node.query( """ CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; """ ) test_rocksdb_options/test.py:178: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3603: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:34 [ 334 ] DEBUG : Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; on node (cluster.py:3602, query) ___________________________ test_table_valid_options ___________________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_table_valid_options(start_cluster): > node.query( """ CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; """ ) test_rocksdb_options/test.py:85: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3603: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:36 [ 334 ] DEBUG : Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; on node (cluster.py:3602, query) _______________________ test_valid_column_family_options _______________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_valid_column_family_options(start_cluster): > node.query( """ CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; """ ) test_rocksdb_options/test.py:119: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3603: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:37 [ 334 ] DEBUG : Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; on node (cluster.py:3602, query) ______________________________ test_valid_options ______________________________ [gw2] linux -- Python 3.10.12 /usr/bin/python3 start_cluster = def test_valid_options(start_cluster): > node.query( """ CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; """ ) test_rocksdb_options/test.py:27: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3603: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.3.2:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:39 [ 334 ] DEBUG : Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB PRIMARY KEY(key); DROP TABLE test; on node (cluster.py:3602, query) ---------------------------- Captured log teardown ----------------------------- 2026-04-02 10:41:43 [ 334 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/.env', '--project-name', 'roottestrocksdboptions_gw2', '--file', '/ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-04-02 10:41:51 [ 334 ] DEBUG : Stderr:Stopping roottestrocksdboptions_gw2_node_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:41:51 [ 334 ] DEBUG : Stderr:Stopping roottestrocksdboptions_gw2_node_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:41:51 [ 334 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-04-02 10:41:52 [ 334 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/.env', '--project-name', 'roottestrocksdboptions_gw2', '--file', '/ClickHouse/tests/integration/test_rocksdb_options/_instances_0_gw2/node/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Stderr:Removing roottestrocksdboptions_gw2_node_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Stderr:Removing roottestrocksdboptions_gw2_node_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Stderr:Removing network roottestrocksdboptions_gw2_default (cluster.py:123, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Docker networks for project roottestrocksdboptions_gw2 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:41:57 [ 334 ] DEBUG : Docker containers for project roottestrocksdboptions_gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:41:57 [ 334 ] DEBUG : Docker volumes for project roottestrocksdboptions_gw2 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:41:57 [ 334 ] DEBUG : Command:docker container list --all --filter name='^/roottestrocksdboptions_gw2_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : No running containers for project: roottestrocksdboptions_gw2 (cluster.py:904, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:41:57 [ 334 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:41:57 [ 334 ] DEBUG : Stdout:3 (cluster.py:121, run_and_check) ____________________________ test_single_hudi_file _____________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_single_hudi_file(started_cluster): instance = started_cluster.instances["node1"] spark = started_cluster.spark_session minio_client = started_cluster.minio_client bucket = started_cluster.minio_bucket TABLE_NAME = "test_single_hudi_file" inserted_data = "SELECT number as a, toString(number) as b FROM numbers(100)" parquet_data_path = create_initial_data_file( started_cluster, instance, inserted_data, TABLE_NAME ) > write_hudi_from_file(spark, TABLE_NAME, parquet_data_path, f"/{TABLE_NAME}") test_storage_hudi/test.py:177: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_storage_hudi/test.py:116: in write_hudi_from_file spark.conf.set("spark.sql.debug.maxToStringFields", 100000) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = key = 'spark.sql.debug.maxToStringFields', value = 100000 @since(2.0) def set(self, key: str, value: str) -> None: """Sets the given Spark runtime configuration property.""" > self._jconf.set(key, value) E AttributeError: 'NoneType' object has no attribute 'set' /usr/local/lib/python3.10/dist-packages/pyspark/sql/conf.py:40: AttributeError ----------------------------- Captured stdout call ----------------------------- 26/04/02 10:41:41 WARN HoodieWriteConfig: Embedded timeline server is disabled, fallback to use direct marker type for spark ----------------------------- Captured stderr call ----------------------------- [Stage 143:> (0 + 0) / 9] [Stage 143:> (0 + 1) / 9] [Stage 143:======> (1 + 1) / 9] [Stage 143:============> (2 + 1) / 9] [Stage 143:===================> (3 + 1) / 9] [Stage 143:=========================> (4 + 1) / 9] [Stage 143:===============================> (5 + 1) / 9] [Stage 143:======================================> (6 + 1) / 9] [Stage 143:============================================> (7 + 1) / 9] [Stage 143:==================================================> (8 + 1) / 9] [Stage 148:> (0 + 1) / 9] [Stage 148:======> (1 + 1) / 9] [Stage 148:============> (2 + 1) / 9] [Stage 148:===================> (3 + 1) / 9] [Stage 148:=========================> (4 + 1) / 9] [Stage 148:===============================> (5 + 2) / 9] [Stage 148:============================================> (7 + 1) / 9] [Stage 148:==================================================> (8 + 1) / 9] ------------------------------ Captured log call ------------------------------- 2026-04-02 10:36:57 [ 409 ] DEBUG : Executing query INSERT INTO TABLE FUNCTION file('test_single_hudi_file.parquet') SETTINGS output_format_parquet_compression_method='none', s3_truncate_on_insert=1 SELECT number as a, toString(number) as b FROM numbers(100) FORMAT Parquet on node1 (cluster.py:3602, query) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o298 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o299 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o301 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o302 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o303 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o304 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o306 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o307 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o309 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o310 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o311 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o312 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o313 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o314 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o315 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o316 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o317 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o318 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o319 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o320 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o321 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o322 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o323 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o244 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o251 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o254 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o232 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o290 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Command to send: m d o297 e (clientserver.py:501, send_command) 2026-04-02 10:36:57 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:36:59 [ 409 ] DEBUG : Command to send: c o59 conf e (clientserver.py:501, send_command) 2026-04-02 10:41:49 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) __________________________________ test_types __________________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_types(started_cluster): instance = started_cluster.instances["node1"] spark = started_cluster.spark_session minio_client = started_cluster.minio_client bucket = started_cluster.minio_bucket TABLE_NAME = "test_types" data = [ ( 123, "string", datetime.strptime("2000-01-01", "%Y-%m-%d"), ["str1", "str2"], True, ) ] schema = StructType( [ StructField("a", IntegerType()), StructField("b", StringType()), StructField("c", DateType()), StructField("d", ArrayType(StringType())), StructField("e", BooleanType()), ] ) > df = spark.createDataFrame(data=data, schema=schema) test_storage_hudi/test.py:283: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/local/lib/python3.10/dist-packages/pyspark/sql/session.py:873: in createDataFrame self._jvm.SparkSession.setActiveSession(self._jsparkSession) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = name = 'SparkSession' def __getattr__(self, name): if name == UserHelpAutoCompletion.KEY: return UserHelpAutoCompletion() answer = self._gateway_client.send_command( proto.REFLECTION_COMMAND_NAME + proto.REFL_GET_UNKNOWN_SUB_COMMAND_NAME + name + "\n" + self._id + "\n" + proto.END_COMMAND_PART) if answer == proto.SUCCESS_PACKAGE: return JavaPackage(name, self._gateway_client, jvm_id=self._id) elif answer.startswith(proto.SUCCESS_CLASS): return JavaClass( answer[proto.CLASS_FQN_START:], self._gateway_client) else: _, error_message = get_error_message(answer) message = compute_exception_message( "{0} does not exist in the JVM".format(name), error_message) > raise Py4JError(message) E py4j.protocol.Py4JError: SparkSession does not exist in the JVM /usr/local/lib/python3.10/dist-packages/py4j/java_gateway.py:1722: Py4JError ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:50 [ 409 ] DEBUG : Command to send: r u SparkSession rj e (clientserver.py:501, send_command) 2026-04-02 10:41:50 [ 409 ] DEBUG : Answer received: !yro325 (clientserver.py:512, send_command) ---------------------------- Captured log teardown ----------------------------- 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o186 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o170 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o184 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o185 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o179 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o308 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Command to send: m d o324 e (clientserver.py:501, send_command) 2026-04-02 10:41:52 [ 409 ] DEBUG : Answer received: !yv (clientserver.py:512, send_command) 2026-04-02 10:41:53 [ 409 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_minio1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_resolver_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_proxy2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_proxy1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_minio1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_resolver_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Stderr:Stopping rootteststoragehudi_gw7_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-04-02 10:42:19 [ 409 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/.env', '--project-name', 'rootteststoragehudi_gw7', '--file', '/ClickHouse/tests/integration/test_storage_hudi/_instances_0_gw7/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_minio1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_resolver_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_proxy2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_proxy1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_resolver_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_minio1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_proxy2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing rootteststoragehudi_gw7_proxy1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing network rootteststoragehudi_gw7_default (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Stderr:Removing volume rootteststoragehudi_gw7_data1-1 (cluster.py:123, run_and_check) 2026-04-02 10:42:22 [ 409 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Docker networks for project rootteststoragehudi_gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:42:23 [ 409 ] DEBUG : Docker containers for project rootteststoragehudi_gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:42:23 [ 409 ] DEBUG : Docker volumes for project rootteststoragehudi_gw7 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:42:23 [ 409 ] DEBUG : Command:docker container list --all --filter name='^/rootteststoragehudi_gw7_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:42:23 [ 409 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : No running containers for project: rootteststoragehudi_gw7 (cluster.py:904, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:42:23 [ 409 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:42:23 [ 409 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:42:23 [ 409 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:42:23 [ 409 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) _______________________________ test_update_disk _______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_disk(started_cluster): try: name = "test_update_disk" engine = "MergeTree()" start_over() > node1.restart_clickhouse(kill=True) test_reloading_storage_configuration/test.py:207: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-02 10:41:24 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:24 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:41:24 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:41:24 [ 328 ] DEBUG : Stdout: 6915 ? 00:00:37 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:41:24 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:24 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'pkill -9 clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:41:25 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:25 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:25 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:26 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:26 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:27 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:28 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:28 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:29 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:30 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:30 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:31 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:32 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:32 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:32 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:33 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:33 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:34 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:35 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:35 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:36 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:37 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:37 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:37 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:38 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:39 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:40 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:40 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:41 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:42 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:42 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:43 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:44 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:44 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:44 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:45 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:45 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:46 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:47 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:47 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:48 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:49 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:49 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:50 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:51 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:51 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:52 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:53 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:53 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:53 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:54 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:54 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:55 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:56 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:56 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:57 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:58 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:58 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:41:58 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:41:59 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:41:59 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:00 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:01 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:01 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:01 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:02 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:02 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:03 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:04 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:04 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:04 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:05 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:05 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:06 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:07 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:07 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:07 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:08 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:08 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:09 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:10 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:10 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:10 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:11 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:11 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:12 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:13 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:13 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:13 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:14 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:14 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:15 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:16 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:16 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:16 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:17 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:17 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:18 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:19 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:19 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:19 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:20 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:20 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:21 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:22 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:22 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:22 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:23 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:23 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:24 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:25 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:25 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:26 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:26 [ 328 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:6915 (cluster.py:3926, stop_clickhouse) 2026-04-02 10:42:26 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:26 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-02 10:42:26 [ 328 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-02 10:42:26 [ 328 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-02 10:42:26 [ 328 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915 > /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-02 10:42:26 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:26 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:27 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:27 [ 328 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-02 10:42:27 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:27 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:27 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:27 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:28 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:29 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:30 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:31 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:32 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:34 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:35 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:36 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:37 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:38 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:38 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:38 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:42:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:38 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:39 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:39 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:40 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:41 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:42 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:43 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:44 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:46 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:47 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:48 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:49 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:51 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:51 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:51 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:51 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:42:51 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:42:51 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:42:52 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:42:52 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:53 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:54 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:55 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:56 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:57 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:58 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:42:59 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:01 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:02 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:03 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:03 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:03 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:03 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:43:03 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:03 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:03 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:03 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:05 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:06 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:07 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:08 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:09 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:10 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:11 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:12 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:13 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:14 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:14 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:15 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:15 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:43:15 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:15 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:15 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:15 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:16 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:17 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:18 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:19 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:20 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:21 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:22 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:23 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:24 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:25 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:25 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:26 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:26 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:43:26 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:26 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:26 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:26 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:27 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:28 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:29 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:30 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:31 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:32 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:33 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:34 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:35 [ 328 ] DEBUG : Executing query select 20 on node1 (cluster.py:3602, query) 2026-04-02 10:43:36 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:36 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:37 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:37 [ 328 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.6:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-02 10:43:37 [ 328 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-02 10:43:37 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:37 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-02 10:43:37 [ 328 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-02 10:43:37 [ 328 ] DEBUG : Stdout: 6915 ? 00:00:37 clickhouse (cluster.py:121, run_and_check) 2026-04-02 10:43:37 [ 328 ] INFO : PS RESULT: PID TTY TIME CMD 6915 ? 00:00:37 clickhouse (cluster.py:4019, wait_start) 2026-04-02 10:43:37 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:37 [ 328 ] DEBUG : Command:['docker', 'exec', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-02 10:43:38 [ 328 ] DEBUG : Stdout:6915 (cluster.py:121, run_and_check) 2026-04-02 10:43:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:38 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 6915"] (cluster.py:113, run_and_check) 2026-04-02 10:43:38 [ 328 ] DEBUG : Stderr:warning: process 6915 is already traced by process 7673process 6915 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-02 10:43:38 [ 328 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-02 10:43:38 [ 328 ] WARNING : Current start attempt failed. Will kill 6915 just in case. (cluster.py:3982, start_clickhouse) 2026-04-02 10:43:38 [ 328 ] DEBUG : run container_id:roottestreloadingstorageconfiguration_gw0_node1_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 6915'] (cluster.py:2173, exec_in_container) 2026-04-02 10:43:38 [ 328 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestreloadingstorageconfiguration_gw0_node1_1', 'bash', '-c', 'kill -9 6915'] (cluster.py:113, run_and_check) 2026-04-02 10:43:39 [ 328 ] DEBUG : Executing query DROP TABLE IF EXISTS test_update_disk on node1 (cluster.py:3602, query) ---------------------------- Captured log teardown ----------------------------- 2026-04-02 10:43:42 [ 328 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/.env', '--project-name', 'roottestreloadingstorageconfiguration_gw0', '--file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_node2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_node2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Stderr:Stopping roottestreloadingstorageconfiguration_gw0_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-04-02 10:43:49 [ 328 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/.env', '--project-name', 'roottestreloadingstorageconfiguration_gw0', '--file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_reloading_storage_configuration/_instances_0_gw0/node2/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_node1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_node2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo1_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo3_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo2_1 ... (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_node2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo2_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_zoo3_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing roottestreloadingstorageconfiguration_gw0_node1_1 ... done (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Stderr:Removing network roottestreloadingstorageconfiguration_gw0_default (cluster.py:123, run_and_check) 2026-04-02 10:43:51 [ 328 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-02 10:43:51 [ 328 ] DEBUG : Docker networks for project roottestreloadingstorageconfiguration_gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-02 10:43:52 [ 328 ] DEBUG : Docker containers for project roottestreloadingstorageconfiguration_gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-02 10:43:52 [ 328 ] DEBUG : Docker volumes for project roottestreloadingstorageconfiguration_gw0 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-02 10:43:52 [ 328 ] DEBUG : Command:docker container list --all --filter name='^/roottestreloadingstorageconfiguration_gw0_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-02 10:43:52 [ 328 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : No running containers for project: roottestreloadingstorageconfiguration_gw0 (cluster.py:904, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-02 10:43:52 [ 328 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:121, run_and_check) 2026-04-02 10:43:52 [ 328 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-02 10:43:52 [ 328 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-02 10:43:52 [ 328 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) =============================== warnings summary =============================== test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy /usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.16.8.4'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 810.25s call test_storage_hudi/test.py::test_multiple_hudi_files 497.39s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 436.29s setup test_storage_s3_queue/test.py::test_commit_on_limit 424.37s call test_recovery_replica/test.py::test_update_metadata 412.21s call test_rocksdb_options/test.py::test_invalid_column_family_options 397.69s call test_reloading_storage_configuration/test.py::test_remove_policy 292.53s call test_storage_hudi/test.py::test_single_hudi_file 253.72s call test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 204.00s call test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 173.73s call test_settings_constraints_distributed/test.py::test_select_clamps_settings 163.56s call test_recovery_replica/test.py::test_choose_source_replica 160.99s call test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 153.19s call test_rocksdb_options/test.py::test_invalid_options 152.35s setup test_replace_partition/test.py::test_normal_work 140.56s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 140.05s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 138.87s call test_rocksdb_options/test.py::test_table_invalid_options 135.59s call test_reloading_storage_configuration/test.py::test_update_disk 129.23s call test_rocksdb_options/test.py::test_invalid_block_based_table_options 118.84s call test_session_log/test.py::test_parallel_sessions 117.94s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 109.92s setup test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 108.18s call test_recovery_replica/test.py::test_recovery 106.71s setup test_settings_constraints_distributed/test.py::test_insert_clamps_settings 103.69s call test_row_policy/test.py::test_grant_create_row_policy 99.86s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 96.79s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 93.92s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 90.02s call test_reload_zookeeper/test.py::test_reload_zookeeper 89.83s setup test_storage_hudi/test.py::test_multiple_hudi_files 89.66s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 89.27s setup test_recovery_replica/test.py::test_choose_source_replica 88.65s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy 84.73s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 83.11s call test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 80.10s setup test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl 79.82s call test_reloading_storage_configuration/test.py::test_add_policy 77.30s setup test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 73.70s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 72.57s call test_row_policy/test.py::test_dcl_introspection 68.68s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy 65.24s teardown test_recovery_replica/test.py::test_update_metadata 65.05s setup test_reload_zookeeper/test.py::test_reload_zookeeper 64.21s teardown test_row_policy/test.py::test_grant_create_row_policy 63.17s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 62.34s call test_storage_s3_queue/test.py::test_commit_on_limit 60.73s call test_reloading_storage_configuration/test.py::test_remove_disk 60.21s teardown test_replace_partition/test.py::test_normal_work 59.98s teardown test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 59.76s setup test_settings_constraints_distributed/test.py::test_select_clamps_settings 59.50s call test_session_log/test.py::test_postgres_session 59.41s setup test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 57.94s setup test_storage_redis/test.py::test_create_table 57.79s setup test_reloading_storage_configuration/test.py::test_add_disk 57.77s call test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 57.67s setup test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 56.38s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy 56.10s setup test_part_uuid/test.py::test_part_uuid 55.48s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 55.12s call test_reloading_storage_configuration/test.py::test_new_policy_works 55.05s setup test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 54.08s setup test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 52.78s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 51.98s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 51.14s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 50.15s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy 50.08s setup test_session_log/test.py::test_grpc_session 49.45s teardown test_storage_redis/test.py::test_update 49.37s teardown test_row_policy/test.py::test_introspection 48.37s call test_replace_partition/test.py::test_normal_work 47.08s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 46.61s teardown test_session_log/test.py::test_postgres_session 45.87s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 45.72s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 45.42s setup test_read_temporary_tables_on_failure/test.py::test_different_versions 44.17s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 42.74s call test_reloading_storage_configuration/test.py::test_add_disk 42.48s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 40.93s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 40.78s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 40.59s teardown test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] 40.36s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy 39.74s call test_row_policy/test.py::test_dcl_management 39.32s call test_row_policy/test.py::test_smoke 39.00s call test_storage_redis/test.py::test_simple_insert 38.95s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy 37.48s teardown test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl 37.26s teardown test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 35.22s teardown test_settings_constraints_distributed/test.py::test_select_clamps_settings 34.93s call test_row_policy/test.py::test_miscellaneous_engines 34.66s setup test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side 34.40s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 33.42s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list 33.15s setup test_rocksdb_options/test.py::test_invalid_block_based_table_options 32.52s teardown test_storage_hudi/test.py::test_types 32.38s call test_session_log/test.py::test_grpc_session 32.27s call test_row_policy/test.py::test_some_users_without_policies 31.72s teardown test_reload_zookeeper/test.py::test_reload_zookeeper 31.61s call test_part_uuid/test.py::test_part_uuid 30.28s teardown test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 29.97s teardown test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 29.81s teardown test_read_temporary_tables_on_failure/test.py::test_different_versions 29.67s teardown test_part_uuid/test.py::test_part_uuid 29.01s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 28.12s call test_row_policy/test.py::test_reload_users_xml_by_timer 27.90s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 27.81s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 27.60s call test_settings_constraints_distributed/test.py::test_insert_clamps_settings 26.84s teardown test_row_policy/test.py::test_dcl_management 25.83s teardown test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 25.23s teardown test_row_policy/test.py::test_with_prewhere 24.84s call test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy 24.67s teardown test_row_policy/test.py::test_dcl_introspection 23.91s teardown test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 22.41s call test_storage_redis/test.py::test_simple_select 21.82s teardown test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 21.78s teardown test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 21.77s call test_session_log/test.py::test_mysql_session 21.45s call test_storage_redis/test.py::test_truncate 21.02s call test_storage_redis/test.py::test_create_table 20.92s call test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 20.87s call test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] 20.81s call test_row_policy/test.py::test_row_policy_filter_with_subquery 20.49s teardown test_row_policy/test.py::test_row_policy_filter_with_subquery 20.30s call test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] 19.24s teardown test_row_policy/test.py::test_smoke 18.84s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 18.74s teardown test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side 18.16s call test_storage_redis/test.py::test_update 16.93s call test_row_policy/test.py::test_with_prewhere 16.69s teardown test_rocksdb_options/test.py::test_valid_options 16.51s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 16.26s call test_read_temporary_tables_on_failure/test.py::test_different_versions 16.13s teardown test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 16.06s teardown test_row_policy/test.py::test_join 14.89s call test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 14.72s teardown test_row_policy/test.py::test_reload_users_xml_by_timer 14.66s teardown test_row_policy/test.py::test_miscellaneous_engines 14.63s teardown test_row_policy/test.py::test_policy_on_distributed_table_via_role 14.40s call test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl 13.30s teardown test_row_policy/test.py::test_some_users_without_policies 13.16s call test_row_policy/test.py::test_tags_with_db_and_table_names 12.90s teardown test_row_policy/test.py::test_tags_with_db_and_table_names 12.48s call test_row_policy/test.py::test_policy_on_distributed_table_via_role 12.38s teardown test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter 12.38s teardown test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter 12.37s teardown test_reloading_storage_configuration/test.py::test_update_disk 11.49s call test_storage_redis/test.py::test_delete 10.02s teardown test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data 9.62s call test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data 9.42s call test_storage_redis/test.py::test_select_int 8.43s call test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side 8.27s teardown test_row_policy/test.py::test_users_xml_is_readonly 7.87s call test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data 7.47s teardown test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data 7.40s call test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 6.97s call test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 6.40s call test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 6.18s call test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter 5.69s call test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter 4.40s call test_row_policy/test.py::test_introspection 4.35s call test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 3.39s call test_row_policy/test.py::test_join 3.35s setup test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] 3.30s setup test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] 2.74s call test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage 1.53s call test_rocksdb_options/test.py::test_valid_column_family_options 1.11s call test_row_policy/test.py::test_users_xml_is_readonly 0.79s call test_rocksdb_options/test.py::test_valid_options 0.76s call test_rocksdb_options/test.py::test_table_valid_options 0.61s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.44s setup test_recovery_replica/test.py::test_recovery 0.21s teardown test_recovery_replica/test.py::test_choose_source_replica 0.12s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.11s call test_storage_hudi/test.py::test_types 0.07s teardown test_storage_s3_queue/test.py::test_commit_on_limit 0.07s setup test_row_policy/test.py::test_dcl_introspection 0.07s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.06s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list 0.05s setup test_storage_hudi/test.py::test_types 0.04s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 0.04s teardown test_storage_redis/test.py::test_create_table 0.03s setup test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 0.03s teardown test_storage_hudi/test.py::test_multiple_hudi_files 0.03s setup test_storage_redis/test.py::test_update 0.03s setup test_storage_redis/test.py::test_delete 0.03s teardown test_session_log/test.py::test_mysql_session 0.03s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 0.03s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.03s setup test_row_policy/test.py::test_tags_with_db_and_table_names 0.02s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.02s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.02s teardown test_storage_hudi/test.py::test_single_hudi_file 0.02s teardown test_reloading_storage_configuration/test.py::test_new_policy_works 0.02s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 0.02s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.02s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.02s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 0.02s setup test_session_log/test.py::test_parallel_sessions 0.02s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.02s setup test_storage_redis/test.py::test_simple_select 0.02s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.02s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.01s setup test_session_log/test.py::test_postgres_session 0.01s setup test_row_policy/test.py::test_smoke 0.01s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy 0.01s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.01s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.01s setup test_row_policy/test.py::test_users_xml_is_readonly 0.01s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.01s setup test_row_policy/test.py::test_grant_create_row_policy 0.01s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list 0.01s teardown test_rocksdb_options/test.py::test_invalid_block_based_table_options 0.01s setup test_row_policy/test.py::test_introspection 0.01s setup test_recovery_replica/test.py::test_update_metadata 0.01s setup test_row_policy/test.py::test_dcl_management 0.01s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.01s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.01s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.01s setup test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data 0.01s setup test_rocksdb_options/test.py::test_invalid_column_family_options 0.01s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy 0.01s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s setup test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s setup test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter 0.00s teardown test_settings_constraints_distributed/test.py::test_insert_clamps_settings 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s teardown test_session_log/test.py::test_parallel_sessions 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_storage_hudi/test.py::test_single_hudi_file 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.00s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy 0.00s setup test_row_policy/test.py::test_with_prewhere 0.00s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy 0.00s setup test_row_policy/test.py::test_reload_users_xml_by_timer 0.00s setup test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data 0.00s setup test_row_policy/test.py::test_some_users_without_policies 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s setup test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_row_policy/test.py::test_join 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 0.00s setup test_row_policy/test.py::test_policy_on_distributed_table_via_role 0.00s setup test_row_policy/test.py::test_row_policy_filter_with_subquery 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_session_log/test.py::test_grpc_session 0.00s teardown test_storage_redis/test.py::test_simple_insert 0.00s setup test_storage_redis/test.py::test_select_int 0.00s setup test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 0.00s setup test_storage_redis/test.py::test_truncate 0.00s teardown test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.00s teardown test_storage_redis/test.py::test_simple_select 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s setup test_row_policy/test.py::test_miscellaneous_engines 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s setup test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy 0.00s setup test_session_log/test.py::test_mysql_session 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.00s teardown test_storage_redis/test.py::test_delete 0.00s setup test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_storage_redis/test.py::test_truncate 0.00s setup test_storage_redis/test.py::test_simple_insert 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s teardown test_storage_redis/test.py::test_select_int 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] =========================== short test summary info ============================ FAILED test_rocksdb_options/test.py::test_invalid_column_family_options - Exc... FAILED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] FAILED test_rocksdb_options/test.py::test_invalid_options - Exception: Cannot... FAILED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options FAILED test_storage_hudi/test.py::test_multiple_hudi_files - Failed: Timeout ... FAILED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper FAILED test_reloading_storage_configuration/test.py::test_remove_policy - Exc... FAILED test_rocksdb_options/test.py::test_table_invalid_column_family_options FAILED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy FAILED test_rocksdb_options/test.py::test_table_invalid_options - Exception: ... FAILED test_rocksdb_options/test.py::test_table_valid_column_family_options FAILED test_rocksdb_options/test.py::test_table_valid_options - helpers.clien... FAILED test_rocksdb_options/test.py::test_valid_column_family_options - helpe... FAILED test_rocksdb_options/test.py::test_valid_options - helpers.client.Quer... FAILED test_storage_hudi/test.py::test_single_hudi_file - AttributeError: 'No... FAILED test_storage_hudi/test.py::test_types - py4j.protocol.Py4JError: Spark... FAILED test_reloading_storage_configuration/test.py::test_update_disk - Excep... PASSED test_storage_redis/test.py::test_create_table PASSED test_session_log/test.py::test_grpc_session PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] PASSED test_storage_redis/test.py::test_delete PASSED test_storage_redis/test.py::test_select_int PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_session_log/test.py::test_mysql_session PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy PASSED test_storage_redis/test.py::test_simple_insert PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with PASSED test_storage_redis/test.py::test_simple_select PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy PASSED test_storage_redis/test.py::test_truncate PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_storage_redis/test.py::test_update PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list PASSED test_session_log/test.py::test_parallel_sessions PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] PASSED test_session_log/test.py::test_postgres_session PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] PASSED test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] PASSED test_recovery_replica/test.py::test_recovery PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_row_policy/test.py::test_dcl_introspection PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions PASSED test_row_policy/test.py::test_dcl_management PASSED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml PASSED test_settings_constraints_distributed/test.py::test_insert_clamps_settings PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_row_policy/test.py::test_grant_create_row_policy PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings PASSED test_replace_partition/test.py::test_normal_work PASSED test_row_policy/test.py::test_introspection PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_settings_constraints_distributed/test.py::test_select_clamps_settings PASSED test_storage_s3_queue/test.py::test_commit_on_limit PASSED test_row_policy/test.py::test_join PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] PASSED test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered] PASSED test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered] PASSED test_row_policy/test.py::test_miscellaneous_engines PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] PASSED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned PASSED test_row_policy/test.py::test_policy_on_distributed_table_via_role PASSED test_part_uuid/test.py::test_part_uuid PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] PASSED test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] PASSED test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side PASSED test_row_policy/test.py::test_reload_users_xml_by_timer PASSED test_row_policy/test.py::test_row_policy_filter_with_subquery PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] PASSED test_row_policy/test.py::test_smoke PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format PASSED test_reload_zookeeper/test.py::test_reload_zookeeper PASSED test_row_policy/test.py::test_some_users_without_policies PASSED test_row_policy/test.py::test_tags_with_db_and_table_names PASSED test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter PASSED test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter PASSED test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data PASSED test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data PASSED test_row_policy/test.py::test_users_xml_is_readonly PASSED test_row_policy/test.py::test_with_prewhere SKIPPED [1] test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py:29: Disabled for debug and sanitizers. Too slow. ======= 17 failed, 82 passed, 1 skipped, 1 warning in 1333.66s (0:22:13) ======= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 445, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_6biipx --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_merge_tree_replicated_db_ttl/test.py::test_replicated_db_and_ttl test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke test_row_policy/test.py::test_some_users_without_policies test_row_policy/test.py::test_tags_with_db_and_table_names test_row_policy/test.py::test_throwif_error_in_prewhere_with_same_condition_as_filter test_row_policy/test.py::test_throwif_error_in_where_with_same_condition_as_filter test_row_policy/test.py::test_throwif_in_prewhere_doesnt_expose_restricted_data test_row_policy/test.py::test_throwif_in_where_doesnt_expose_restricted_data test_row_policy/test.py::test_users_xml_is_readonly test_row_policy/test.py::test_with_prewhere test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_env_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_proxy_list_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_no_proxy test_s3_table_function_with_https_proxy/test.py::test_s3_with_https_remote_proxy test_s3_zero_copy_ttl/test_vertical_merge_memory_usage.py::test_vertical_merge_memory_usage test_server_keep_alive/test.py::test_max_keep_alive_requests_on_user_side test_session_log/test.py::test_grpc_session test_session_log/test.py::test_mysql_session test_session_log/test.py::test_parallel_sessions test_session_log/test.py::test_postgres_session test_settings_constraints_distributed/test.py::test_insert_clamps_settings test_settings_constraints_distributed/test.py::test_select_clamps_settings test_storage_hudi/test.py::test_multiple_hudi_files test_storage_hudi/test.py::test_single_hudi_file test_storage_hudi/test.py::test_types test_storage_redis/test.py::test_create_table test_storage_redis/test.py::test_delete test_storage_redis/test.py::test_select_int test_storage_redis/test.py::test_simple_insert test_storage_redis/test.py::test_simple_select test_storage_redis/test.py::test_truncate test_storage_redis/test.py::test_update test_storage_s3_queue/test.py::test_commit_on_limit 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-ordered]' 'test_storage_s3_queue/test.py::test_delete_after_processing[AzureQueue-unordered]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa ' returned non-zero exit status 1.